Jul 27 21:45:15.074529 ip-10-0-138-124 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 21:45:15.459942 ip-10-0-138-124 kubenswrapper[2549]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:45:15.459942 ip-10-0-138-124 kubenswrapper[2549]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 21:45:15.459942 ip-10-0-138-124 kubenswrapper[2549]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:45:15.459942 ip-10-0-138-124 kubenswrapper[2549]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 21:45:15.459942 ip-10-0-138-124 kubenswrapper[2549]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:45:15.461542 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.461422    2549 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 21:45:15.466630 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466613    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466632    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466637    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466641    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466644    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466647    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466650    2549 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466653    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466656    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466658    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466661    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466664    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466667    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466670    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466673    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:45:15.466669 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466677    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466681    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466684    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466687    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466690    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466692    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466695    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466699    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466701    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466705    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466708    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466711    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466713    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466717    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466719    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466723    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466725    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466729    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466731    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:45:15.467034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466734    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466736    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466738    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466741    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466743    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466746    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466748    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466750    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466753    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466756    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466758    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466761    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466763    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466766    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466768    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466772    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466775    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466777    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466780    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466783    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:45:15.467503 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466785    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466788    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466790    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466793    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466795    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466798    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466800    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466803    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466805    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466808    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466810    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466813    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466815    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466817    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466820    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466822    2549 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466824    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466827    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466830    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466832    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:45:15.468000 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466834    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466837    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466839    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466842    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466844    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466846    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466850    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466854    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466856    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466859    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466861    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466864    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466866    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466868    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466871    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466873    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466876    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466878    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466880    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466883    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:45:15.468495 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466885    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466889    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466893    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466896    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.466898    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467609    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467615    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467619    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467622    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467625    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467628    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467630    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467633    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467636    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467639    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467641    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467644    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467646    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467649    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467652    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:45:15.468980 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467655    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467657    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467660    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467662    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467665    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467667    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467669    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467672    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467674    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467678    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467680    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467683    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467685    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467688    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467690    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467693    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467695    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467698    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467700    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467703    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:45:15.469491 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467705    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467708    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467710    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467713    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467715    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467718    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467720    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467723    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467725    2549 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467728    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467730    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467734    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467736    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467739    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467741    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467744    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467746    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467748    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467751    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467753    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:45:15.469975 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467756    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467758    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467762    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467764    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467767    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467769    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467771    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467774    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467776    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467778    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467781    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467783    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467786    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467788    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467791    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467793    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467796    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467799    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467801    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:45:15.470593 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467803    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467806    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467808    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467811    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467814    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467817    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467819    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467822    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467824    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467827    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467829    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467832    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467834    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467837    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467840    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467843    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467847    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467850    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467853    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467856    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:45:15.471045 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467859    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467862    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467865    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467869    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.467872    2549 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468289    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468296    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468299    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468301    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468305    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468307    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468310    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468313    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468316    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468318    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468321    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468327    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468330    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468332    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:45:15.471546 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468335    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468337    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468340    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468343    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468345    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468348    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468351    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468354    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468357    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468359    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468362    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468365    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468368    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468370    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468372    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468375    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468377    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468380    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468382    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:45:15.472039 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468385    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468388    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468390    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468392    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468396    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468399    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468401    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468404    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468406    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468408    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468412    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468415    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468417    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468420    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468423    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468425    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468428    2549 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468430    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468433    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468435    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:45:15.472516 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468437    2549 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468442    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468445    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468448    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468451    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468454    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468457    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468460    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468462    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468465    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468467    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468470    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468475    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468478    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468481    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468483    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468486    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468489    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468492    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:45:15.473009 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468494    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468497    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468499    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468503    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468506    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468508    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468511    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468513    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468516    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468518    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468521    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468523    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468526    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468528    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468531    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468533    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468536    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468538    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468540    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468543    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:45:15.473493 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468545    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468548    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468550    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468552    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468555    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468557    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.468560    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468632    2549 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468643    2549 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468653    2549 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468659    2549 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468666    2549 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468671    2549 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468676    2549 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468681    2549 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468684    2549 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468687    2549 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468691    2549 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468694    2549 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468697    2549 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468700    2549 flags.go:64] FLAG: --cgroup-root=""
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468703    2549 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468706    2549 flags.go:64] FLAG: --client-ca-file=""
Jul 27 21:45:15.473978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468709    2549 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468712    2549 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468716    2549 flags.go:64] FLAG: --cluster-domain=""
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468719    2549 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468722    2549 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468725    2549 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468728    2549 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468732    2549 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468734    2549 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468740    2549 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468743    2549 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468746    2549 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468749    2549 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468752    2549 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468755    2549 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468758    2549 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468762    2549 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468766    2549 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468769    2549 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468771    2549 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468774    2549 flags.go:64] FLAG: --enable-server="true"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468778    2549 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468783    2549 flags.go:64] FLAG: --event-burst="100"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468786    2549 flags.go:64] FLAG: --event-qps="50"
Jul 27 21:45:15.474539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468789    2549 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468792    2549 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468795    2549 flags.go:64] FLAG: --eviction-hard=""
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468798    2549 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468801    2549 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468804    2549 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468807    2549 flags.go:64] FLAG: --eviction-soft=""
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468810    2549 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468813    2549 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468816    2549 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468818    2549 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468821    2549 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468824    2549 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468827    2549 flags.go:64] FLAG: --feature-gates=""
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468831    2549 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468834    2549 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468838    2549 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468842    2549 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468845    2549 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468848    2549 flags.go:64] FLAG: --help="false"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468852    2549 flags.go:64] FLAG: --hostname-override="ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468855    2549 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468857    2549 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 21:45:15.475094 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468872    2549 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468876    2549 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468879    2549 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468882    2549 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468885    2549 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468888    2549 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468891    2549 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468894    2549 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468897    2549 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468900    2549 flags.go:64] FLAG: --kube-reserved=""
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468903    2549 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468906    2549 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468908    2549 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468911    2549 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468914    2549 flags.go:64] FLAG: --lock-file=""
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468916    2549 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468919    2549 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468922    2549 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468927    2549 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468930    2549 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468933    2549 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468935    2549 flags.go:64] FLAG: --logging-format="text"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468938    2549 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468941    2549 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 21:45:15.475698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468944    2549 flags.go:64] FLAG: --manifest-url=""
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468946    2549 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468955    2549 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468958    2549 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468963    2549 flags.go:64] FLAG: --max-pods="110"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468965    2549 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468968    2549 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468971    2549 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468974    2549 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468977    2549 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468980    2549 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468983    2549 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468990    2549 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468993    2549 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468996    2549 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.468998    2549 flags.go:64] FLAG: --pod-cidr=""
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469005    2549 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469011    2549 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469016    2549 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469019    2549 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469022    2549 flags.go:64] FLAG: --port="10250"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469025    2549 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469027    2549 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-047edc736795f99cc"
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469030    2549 flags.go:64] FLAG: --qos-reserved=""
Jul 27 21:45:15.476304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469033    2549 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469036    2549 flags.go:64] FLAG: --register-node="true"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469039    2549 flags.go:64] FLAG: --register-with-taints=""
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469042    2549 flags.go:64] FLAG: --registry-burst="10"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469045    2549 flags.go:64] FLAG: --registry-qps="5"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469048    2549 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469050    2549 flags.go:64] FLAG: --reserved-memory=""
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469054    2549 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469057    2549 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469059    2549 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469064    2549 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469066    2549 flags.go:64] FLAG: --runonce="false"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469069    2549 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469072    2549 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469075    2549 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469078    2549 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469080    2549 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469083    2549 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469086    2549 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469089    2549 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469092    2549 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469095    2549 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469098    2549 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469100    2549 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469103    2549 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 21:45:15.476932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469107    2549 flags.go:64] FLAG: --system-cgroups=""
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469111    2549 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469116    2549 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469119    2549 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469121    2549 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469125    2549 flags.go:64] FLAG: --tls-min-version=""
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469128    2549 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469130    2549 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469133    2549 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469136    2549 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469139    2549 flags.go:64] FLAG: --v="2"
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469142    2549 flags.go:64] FLAG: --version="false"
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469147    2549 flags.go:64] FLAG: --vmodule=""
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469151    2549 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.469154    2549 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469287    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469291    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469296    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469298    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469301    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469304    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469306    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469309    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:45:15.477606 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469312    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469314    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469317    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469319    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469322    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469324    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469327    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469329    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469331    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469334    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469343    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469346    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469348    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469368    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469372    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469375    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469378    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469382    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469386    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469388    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:45:15.478177 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469391    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469394    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469396    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469400    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469402    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469405    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469409    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469411    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469414    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469416    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469419    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469421    2549 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469424    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469426    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469429    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469431    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469434    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469436    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469439    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:45:15.478688 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469442    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469444    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469447    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469451    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469454    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469457    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469459    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469462    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469464    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469467    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469469    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469472    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469474    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469477    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469479    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469482    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469484    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469488    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469491    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469495    2549 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:45:15.479153 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469498    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469500    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469503    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469506    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469508    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469511    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469513    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469516    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469518    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469520    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469523    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469525    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469527    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469530    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469532    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469536    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469539    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469542    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469546    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469549    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:45:15.479670 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469551    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469555    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469557    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469560    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469563    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469565    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469568    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469570    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469573    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469575    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469577    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.469582    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.470402    2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.478759    2549 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.478774    2549 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 21:45:15.480159 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.478794    2549 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.478800    2549 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478856    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478861    2549 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478865    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478868    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478871    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478874    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478876    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478879    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478881    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478884    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478886    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478889    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478891    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478894    2549 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478896    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478898    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478902    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:45:15.480571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478904    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478907    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478909    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478912    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478914    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478917    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478919    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478922    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478924    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478928    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478932    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478935    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478938    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478941    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478943    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478946    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478948    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478951    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478953    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478955    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:45:15.481034 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478958    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478960    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478963    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478965    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478968    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478970    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478973    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478975    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478978    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478980    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478983    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478986    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478989    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478992    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.478996    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479000    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479003    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479006    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479008    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:45:15.481559 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479011    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479013    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479016    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479018    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479021    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479024    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479027    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479030    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479032    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479035    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479038    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479040    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479043    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479045    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479048    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479050    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479053    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479055    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479057    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479060    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:45:15.482062 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479062    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479064    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479067    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479069    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479072    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479074    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479076    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479079    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479082    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479084    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479086    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479089    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479091    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479093    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479095    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479098    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479100    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479103    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479106    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479109    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:45:15.482573 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479111    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479114    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479116    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.479121    2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479256    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479262    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479265    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479268    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479270    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479273    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479276    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479279    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479281    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479284    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479286    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:45:15.483081 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479289    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479291    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479294    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479297    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479299    2549 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479302    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479304    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479306    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479309    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479311    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479314    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479316    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479318    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479321    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479323    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479326    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479329    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479332    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479334    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479337    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:45:15.483625 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479339    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479342    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479344    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479347    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479349    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479352    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479356    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479359    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479362    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479364    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479367    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479370    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479374    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479377    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479380    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479382    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479384    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479387    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479389    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:45:15.484137 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479391    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479394    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479396    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479398    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479401    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479403    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479406    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479408    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479411    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479413    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479416    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479419    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479421    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479424    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479426    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479428    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479431    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479433    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479435    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479438    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:45:15.484626 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479440    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479442    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479445    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479447    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479449    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479452    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479455    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479457    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479460    2549 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479462    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479465    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479467    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479470    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479472    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479474    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479477    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479479    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479481    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479484    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:45:15.485126 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479486    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:45:15.485633 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479489    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:45:15.485633 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479491    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:45:15.485633 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479495    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:45:15.485633 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479498    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:45:15.485633 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479501    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:45:15.485633 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479504    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:45:15.485633 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479506    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:45:15.485633 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479509    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:45:15.485633 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:15.479511    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:45:15.485633 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.479516    2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:45:15.485633 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.479633    2549 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 21:45:15.485633 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.481776    2549 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 21:45:15.485633 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.483058    2549 server.go:1019] "Starting client certificate rotation"
Jul 27 21:45:15.485633 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.483154    2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 21:45:15.485979 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.483189    2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 21:45:15.485979 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.485372    2549 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 21:45:15.491742 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.491719    2549 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 21:45:15.501513 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.501494    2549 log.go:25] "Validated CRI v1 runtime API"
Jul 27 21:45:15.506626 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.506610    2549 log.go:25] "Validated CRI v1 image API"
Jul 27 21:45:15.507506 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.507491    2549 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 21:45:15.510130 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.510105    2549 fs.go:135] Filesystem UUIDs: map[195c2aa7-e472-4e59-99ee-108e6b3c5614:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 da5bed9f-9aa6-4788-8b22-2ca2890240c0:/dev/nvme0n1p3]
Jul 27 21:45:15.510184 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.510133    2549 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 21:45:15.515683 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.515581    2549 manager.go:216] Machine: {Timestamp:2026-07-27 21:45:15.514214426 +0000 UTC m=+0.325561788 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099828 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec269bf939e7d2f4e3158971bcb54291 SystemUUID:ec269bf9-39e7-d2f4-e315-8971bcb54291 BootID:8449dff6-00a5-4141-8c61-300b4546d205 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ab:b5:13:46:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ab:b5:13:46:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:6a:52:c6:84:66 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 21:45:15.515683 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.515681    2549 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 21:45:15.515811 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.515766    2549 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 21:45:15.518512 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.518496    2549 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 21:45:15.519053 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.519027    2549 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 21:45:15.519207 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.519060    2549 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-124.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 21:45:15.519840 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.519829    2549 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 21:45:15.519880 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.519842    2549 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 21:45:15.519880 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.519850    2549 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 21:45:15.520440 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.520430    2549 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 21:45:15.521233 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.521207    2549 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 21:45:15.522760 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.522749    2549 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 21:45:15.522897 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.522887    2549 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 21:45:15.522928 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.522919    2549 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 21:45:15.522954 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.522929    2549 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 21:45:15.522954 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.522940    2549 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 21:45:15.522954 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.522950    2549 kubelet.go:398] "Adding apiserver pod source"
Jul 27 21:45:15.523042 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.522959    2549 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 21:45:15.523365 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.523352    2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 21:45:15.523419 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.523372    2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 21:45:15.530346 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.530327    2549 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 21:45:15.532018 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.532004    2549 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 21:45:15.532067 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.532036    2549 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 21:45:15.533388 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.533368    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 21:45:15.533429 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.533398    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 21:45:15.533429 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.533412    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 21:45:15.533429 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.533423    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 21:45:15.533512 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.533439    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 21:45:15.533512 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.533450    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 21:45:15.533512 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.533466    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 21:45:15.533512 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.533477    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 21:45:15.533512 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.533494    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 21:45:15.533512 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.533507    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 21:45:15.533672 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.533524    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 21:45:15.533672 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.533536    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 21:45:15.533902 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.533885    2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 21:45:15.533938 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.533892    2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 21:45:15.534826 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.534814    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 21:45:15.534826 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.534826    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 21:45:15.538604 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.538586    2549 server.go:1268] "Started kubelet"
Jul 27 21:45:15.538792 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.538741    2549 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 21:45:15.538858 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.538817    2549 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 21:45:15.538916 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.538758    2549 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 21:45:15.539463 ip-10-0-138-124 systemd[1]: Started Kubernetes Kubelet.
Jul 27 21:45:15.540447 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.540431    2549 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 21:45:15.540764 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.540752    2549 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 21:45:15.544065 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.544046    2549 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 21:45:15.544574 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.544557    2549 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 21:45:15.545554 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.544565    2549 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-124.ec2.internal.18c643adaeb62529  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-124.ec2.internal,UID:ip-10-0-138-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-124.ec2.internal,},FirstTimestamp:2026-07-27 21:45:15.538556201 +0000 UTC m=+0.349903563,LastTimestamp:2026-07-27 21:45:15.538556201 +0000 UTC m=+0.349903563,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-124.ec2.internal,}"
Jul 27 21:45:15.547786 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.547774    2549 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 21:45:15.547847 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.547823    2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 21:45:15.547898 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.547870    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-124.ec2.internal\" not found"
Jul 27 21:45:15.548531 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.548512    2549 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 21:45:15.548531 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.548533    2549 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 21:45:15.548669 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.548510    2549 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 21:45:15.548669 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.548630    2549 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 21:45:15.548669 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.548636    2549 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 21:45:15.548785 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.548773    2549 factory.go:153] Registering CRI-O factory
Jul 27 21:45:15.548817 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.548791    2549 factory.go:223] Registration of the crio container factory successfully
Jul 27 21:45:15.548852 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.548841    2549 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 21:45:15.548852 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.548848    2549 factory.go:55] Registering systemd factory
Jul 27 21:45:15.548852 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.548853    2549 factory.go:223] Registration of the systemd container factory successfully
Jul 27 21:45:15.548954 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.548879    2549 factory.go:103] Registering Raw factory
Jul 27 21:45:15.548954 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.548890    2549 manager.go:1196] Started watching for new ooms in manager
Jul 27 21:45:15.549640 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.549563    2549 manager.go:349] Starting recovery of all containers
Jul 27 21:45:15.553248 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.553206    2549 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 21:45:15.553341 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.553269    2549 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 21:45:15.561081 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.561058    2549 manager.go:354] Recovery completed
Jul 27 21:45:15.565931 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.565915    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:45:15.566914 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.566895    2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xbz9c"
Jul 27 21:45:15.567986 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.567972    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:45:15.568056 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.567999    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:45:15.568056 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.568009    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:45:15.568494 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.568480    2549 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 21:45:15.568494 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.568494    2549 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 21:45:15.568585 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.568497    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:45:15.568585 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.568510    2549 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 21:45:15.569269 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.569250    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:45:15.569357 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.569282    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:45:15.569357 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.569291    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:45:15.570090 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.570020    2549 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-124.ec2.internal.18c643adb07733fd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-124.ec2.internal,UID:ip-10-0-138-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-124.ec2.internal,},FirstTimestamp:2026-07-27 21:45:15.567985661 +0000 UTC m=+0.379333026,LastTimestamp:2026-07-27 21:45:15.567985661 +0000 UTC m=+0.379333026,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-124.ec2.internal,}"
Jul 27 21:45:15.570168 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.570143    2549 policy_none.go:49] "None policy: Start"
Jul 27 21:45:15.570168 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.570157    2549 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 21:45:15.570340 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.570170    2549 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 21:45:15.571436 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.571421    2549 policy_none.go:47] "Start"
Jul 27 21:45:15.574089 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.574074    2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xbz9c"
Jul 27 21:45:15.577632 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.577556    2549 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-124.ec2.internal.18c643adb0777c19  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-124.ec2.internal,UID:ip-10-0-138-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-124.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-124.ec2.internal,},FirstTimestamp:2026-07-27 21:45:15.568004121 +0000 UTC m=+0.379351483,LastTimestamp:2026-07-27 21:45:15.568004121 +0000 UTC m=+0.379351483,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-124.ec2.internal,}"
Jul 27 21:45:15.608891 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.608867    2549 manager.go:337] "Starting Device Plugin manager"
Jul 27 21:45:15.609036 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.608904    2549 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 21:45:15.609036 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.608914    2549 server.go:87] "Starting device plugin registration server"
Jul 27 21:45:15.609303 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.609276    2549 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 21:45:15.609355 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.609310    2549 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 21:45:15.609407 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.609390    2549 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 21:45:15.609506 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.609492    2549 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 21:45:15.609562 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.609515    2549 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 21:45:15.609972 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.609954    2549 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 21:45:15.610066 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.610006    2549 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-124.ec2.internal\" not found"
Jul 27 21:45:15.692205 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.692156    2549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 21:45:15.693559 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.693541    2549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 21:45:15.693662 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.693581    2549 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 21:45:15.693662 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.693607    2549 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 21:45:15.693760 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.693649    2549 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 21:45:15.696154 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.696134    2549 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:45:15.710348 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.710273    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:45:15.711316 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.711301    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:45:15.711384 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.711333    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:45:15.711384 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.711345    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:45:15.711384 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.711368    2549 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.720967 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.720945    2549 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.721059 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.720971    2549 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-124.ec2.internal\": node \"ip-10-0-138-124.ec2.internal\" not found"
Jul 27 21:45:15.746014 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.745990    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-124.ec2.internal\" not found"
Jul 27 21:45:15.794171 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.794144    2549 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-124.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal"]
Jul 27 21:45:15.794318 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.794210    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:45:15.795823 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.795806    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:45:15.795923 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.795835    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:45:15.795923 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.795848    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:45:15.797280 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.797266    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:45:15.797357 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.797342    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.797413 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.797380    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:45:15.798015 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.797999    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:45:15.798125 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.798028    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:45:15.798125 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.798039    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:45:15.798125 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.798063    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:45:15.798125 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.798082    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:45:15.798125 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.798091    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:45:15.799205 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.799189    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.799293 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.799214    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:45:15.799893 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.799870    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:45:15.799962 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.799901    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:45:15.799962 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.799912    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:45:15.827935 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.827913    2549 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-124.ec2.internal\" not found" node="ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.832641 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.832625    2549 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-124.ec2.internal\" not found" node="ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.846754 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.846737    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-124.ec2.internal\" not found"
Jul 27 21:45:15.850246 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.850212    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83a6c3c959838c7e0ad7324eda9fbb06-config\") pod \"kube-apiserver-proxy-ip-10-0-138-124.ec2.internal\" (UID: \"83a6c3c959838c7e0ad7324eda9fbb06\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.850309 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.850254    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a02bc3e08ab42c32f5cc4d518b13a237-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal\" (UID: \"a02bc3e08ab42c32f5cc4d518b13a237\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.850309 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.850293    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a02bc3e08ab42c32f5cc4d518b13a237-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal\" (UID: \"a02bc3e08ab42c32f5cc4d518b13a237\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.947258 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:15.947192    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-124.ec2.internal\" not found"
Jul 27 21:45:15.950478 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.950457    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a02bc3e08ab42c32f5cc4d518b13a237-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal\" (UID: \"a02bc3e08ab42c32f5cc4d518b13a237\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.950560 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.950488    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a02bc3e08ab42c32f5cc4d518b13a237-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal\" (UID: \"a02bc3e08ab42c32f5cc4d518b13a237\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.950560 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.950504    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83a6c3c959838c7e0ad7324eda9fbb06-config\") pod \"kube-apiserver-proxy-ip-10-0-138-124.ec2.internal\" (UID: \"83a6c3c959838c7e0ad7324eda9fbb06\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.950560 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.950554    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a02bc3e08ab42c32f5cc4d518b13a237-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal\" (UID: \"a02bc3e08ab42c32f5cc4d518b13a237\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.950654 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.950564    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a02bc3e08ab42c32f5cc4d518b13a237-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal\" (UID: \"a02bc3e08ab42c32f5cc4d518b13a237\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:15.950654 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:15.950613    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83a6c3c959838c7e0ad7324eda9fbb06-config\") pod \"kube-apiserver-proxy-ip-10-0-138-124.ec2.internal\" (UID: \"83a6c3c959838c7e0ad7324eda9fbb06\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:16.047997 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:16.047924    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-124.ec2.internal\" not found"
Jul 27 21:45:16.130434 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:16.130401    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:16.136129 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:16.136105    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:16.148050 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:16.148022    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-124.ec2.internal\" not found"
Jul 27 21:45:16.248449 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:16.248398    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-124.ec2.internal\" not found"
Jul 27 21:45:16.348933 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:16.348842    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-124.ec2.internal\" not found"
Jul 27 21:45:16.449330 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:16.449290    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-124.ec2.internal\" not found"
Jul 27 21:45:16.482457 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:16.482414    2549 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 21:45:16.482925 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:16.482589    2549 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 21:45:16.541533 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:16.541504    2549 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:45:16.549822 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:16.549796    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-124.ec2.internal\" not found"
Jul 27 21:45:16.578498 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:16.578455    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 21:40:16 +0000 UTC" deadline="2028-03-29 22:35:43.905545973 +0000 UTC"
Jul 27 21:45:16.578498 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:16.578493    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14664h50m27.327055916s"
Jul 27 21:45:16.639195 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:16.639165    2549 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:45:16.649030 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:16.648948    2549 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:16.664274 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:16.664246    2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 21:45:16.665463 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:16.665437    2549 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-124.ec2.internal"
Jul 27 21:45:16.674995 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:16.674969    2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 21:45:16.732861 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:16.732810    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02bc3e08ab42c32f5cc4d518b13a237.slice/crio-ba81073d3104af83f94a3f22acd2d7d7150b49b264e39e2a0d41cf3e3340f462 WatchSource:0}: Error finding container ba81073d3104af83f94a3f22acd2d7d7150b49b264e39e2a0d41cf3e3340f462: Status 404 returned error can't find the container with id ba81073d3104af83f94a3f22acd2d7d7150b49b264e39e2a0d41cf3e3340f462
Jul 27 21:45:16.733525 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:16.733499    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a6c3c959838c7e0ad7324eda9fbb06.slice/crio-ecce08552ab33a8499c7a6cd8a47eac3893c53d81cbc1c430f6a25c34aa83b9f WatchSource:0}: Error finding container ecce08552ab33a8499c7a6cd8a47eac3893c53d81cbc1c430f6a25c34aa83b9f: Status 404 returned error can't find the container with id ecce08552ab33a8499c7a6cd8a47eac3893c53d81cbc1c430f6a25c34aa83b9f
Jul 27 21:45:16.737952 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:16.737934    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:45:16.819921 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:16.819886    2549 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:45:17.525843 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.525561    2549 apiserver.go:52] "Watching apiserver"
Jul 27 21:45:17.538192 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.538160    2549 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 21:45:17.538753 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.538721    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-hpzp8","openshift-network-diagnostics/network-check-target-fqsbg","openshift-ovn-kubernetes/ovnkube-node-8ggqh","kube-system/global-pull-secret-syncer-8tk6w","kube-system/konnectivity-agent-zl79r","openshift-cluster-node-tuning-operator/tuned-fwl7s","openshift-dns/node-resolver-fk2vd","openshift-multus/multus-additional-cni-plugins-b9s8g","openshift-network-operator/iptables-alerter-nzf6x","kube-system/kube-apiserver-proxy-ip-10-0-138-124.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z","openshift-image-registry/node-ca-75kmz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal","openshift-multus/multus-hk9dv"]
Jul 27 21:45:17.541443 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.541417    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fk2vd"
Jul 27 21:45:17.543887 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.543856    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.544672 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.544648    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 21:45:17.544933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.544917    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jf6d4\""
Jul 27 21:45:17.545095 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.545082    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 21:45:17.550583 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.548684    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 21:45:17.550583 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.548925    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 21:45:17.550583 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.548978    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.550583 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.549097    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 21:45:17.550583 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.549233    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7lhpd\""
Jul 27 21:45:17.550583 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.549718    2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 21:45:17.550583 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.550269    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:17.550583 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:17.550367    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:17.551399 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.551376    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zl79r"
Jul 27 21:45:17.552073 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.552055    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n79rg\""
Jul 27 21:45:17.552196 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.552104    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 21:45:17.553668 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.553644    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 21:45:17.553988 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.553925    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 21:45:17.554163 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.554142    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 21:45:17.554817 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.554795    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.554887 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.554841    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 21:45:17.555376 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.555359    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6bcsm\""
Jul 27 21:45:17.555499 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.555379    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 21:45:17.555499 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.555488    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 21:45:17.555632 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.555603    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 21:45:17.556676 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.556637    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.558857 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.558837    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:45:17.559171 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.559151    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4p4hl\""
Jul 27 21:45:17.559432 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.559412    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 21:45:17.559508 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.559499    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.559831    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-device-dir\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.559868    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-sys-fs\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.559938    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-systemd-units\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.559986    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-run-netns\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560024    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26ljt\" (UniqueName: \"kubernetes.io/projected/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-kube-api-access-26ljt\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560053    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-run-ovn\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560090    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/67a63f7d-7ca2-4626-abcd-5138b6359eb2-konnectivity-ca\") pod \"konnectivity-agent-zl79r\" (UID: \"67a63f7d-7ca2-4626-abcd-5138b6359eb2\") " pod="kube-system/konnectivity-agent-zl79r"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560125    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-registration-dir\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560155    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-slash\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560182    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-node-log\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560193    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560208    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560246    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73750731-6722-49f8-a9d2-55df538483bf-ovnkube-config\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560276    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73750731-6722-49f8-a9d2-55df538483bf-ovn-node-metrics-cert\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560325    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73750731-6722-49f8-a9d2-55df538483bf-ovnkube-script-lib\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560355    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mttgf\" (UniqueName: \"kubernetes.io/projected/73750731-6722-49f8-a9d2-55df538483bf-kube-api-access-mttgf\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.562279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560382    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/519f6723-a7eb-4bbd-82db-e6664825df3c-tmp-dir\") pod \"node-resolver-fk2vd\" (UID: \"519f6723-a7eb-4bbd-82db-e6664825df3c\") " pod="openshift-dns/node-resolver-fk2vd"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560413    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-run-systemd\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560443    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-run-openvswitch\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560474    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560504    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-cni-netd\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560540    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560565    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-cni-bin\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560598    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4hq2\" (UniqueName: \"kubernetes.io/projected/519f6723-a7eb-4bbd-82db-e6664825df3c-kube-api-access-q4hq2\") pod \"node-resolver-fk2vd\" (UID: \"519f6723-a7eb-4bbd-82db-e6664825df3c\") " pod="openshift-dns/node-resolver-fk2vd"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560631    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-socket-dir\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560661    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-kubelet\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560691    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-etc-openvswitch\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560721    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/519f6723-a7eb-4bbd-82db-e6664825df3c-hosts-file\") pod \"node-resolver-fk2vd\" (UID: \"519f6723-a7eb-4bbd-82db-e6664825df3c\") " pod="openshift-dns/node-resolver-fk2vd"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560755    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-log-socket\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560966    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-dbus\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.560999    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/67a63f7d-7ca2-4626-abcd-5138b6359eb2-agent-certs\") pod \"konnectivity-agent-zl79r\" (UID: \"67a63f7d-7ca2-4626-abcd-5138b6359eb2\") " pod="kube-system/konnectivity-agent-zl79r"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.561156    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.563231 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.561189    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-var-lib-openvswitch\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.564103 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.561219    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73750731-6722-49f8-a9d2-55df538483bf-env-overrides\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.564103 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.561361    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-etc-selinux\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.564103 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.561407    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-kubelet-config\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:17.564103 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.562470    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 21:45:17.564103 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.562595    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qjf94\""
Jul 27 21:45:17.564103 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.562730    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 21:45:17.564103 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.562794    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 21:45:17.567857 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.567595    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lt4gx\""
Jul 27 21:45:17.567857 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.567647    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 21:45:17.567857 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.567706    2549 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 21:45:17.567857 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.567601    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 21:45:17.568093 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.567882    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:17.568093 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:17.567943    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:17.569763 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.569741    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-75kmz"
Jul 27 21:45:17.572205 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.572186    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 21:45:17.572781 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.572762    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:17.572866 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:17.572845    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:17.573493 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.573472    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fccn4\""
Jul 27 21:45:17.573711 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.573695    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 21:45:17.573782 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.573698    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 21:45:17.575379 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.575270    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nzf6x"
Jul 27 21:45:17.579373 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.579350    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:45:17.579475 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.579384    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 21:45:17.579475 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.579411    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g4ffb\""
Jul 27 21:45:17.579598 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.579475    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 21:45:17.594826 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.594804    2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2zzk4"
Jul 27 21:45:17.601760 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.601736    2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2zzk4"
Jul 27 21:45:17.650666 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.650639    2549 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 21:45:17.662103 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.661960    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/519f6723-a7eb-4bbd-82db-e6664825df3c-tmp-dir\") pod \"node-resolver-fk2vd\" (UID: \"519f6723-a7eb-4bbd-82db-e6664825df3c\") " pod="openshift-dns/node-resolver-fk2vd"
Jul 27 21:45:17.662346 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662153    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/519f6723-a7eb-4bbd-82db-e6664825df3c-tmp-dir\") pod \"node-resolver-fk2vd\" (UID: \"519f6723-a7eb-4bbd-82db-e6664825df3c\") " pod="openshift-dns/node-resolver-fk2vd"
Jul 27 21:45:17.662346 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662184    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.662346 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662203    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-cni-netd\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.662346 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662234    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:17.662346 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662265    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/12060402-e3dd-4d2c-8832-11da17b477ec-etc-tuned\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.662346 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662318    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-run-k8s-cni-cncf-io\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.662346 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662346    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/741209f5-9dd3-4529-9966-00a653191015-host-slash\") pod \"iptables-alerter-nzf6x\" (UID: \"741209f5-9dd3-4529-9966-00a653191015\") " pod="openshift-network-operator/iptables-alerter-nzf6x"
Jul 27 21:45:17.662636 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662369    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p52wd\" (UniqueName: \"kubernetes.io/projected/741209f5-9dd3-4529-9966-00a653191015-kube-api-access-p52wd\") pod \"iptables-alerter-nzf6x\" (UID: \"741209f5-9dd3-4529-9966-00a653191015\") " pod="openshift-network-operator/iptables-alerter-nzf6x"
Jul 27 21:45:17.662636 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662388    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-cni-netd\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.662636 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662392    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.662636 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662394    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-cni-bin\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.662636 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662433    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-cni-bin\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.662636 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662483    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-socket-dir\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.662636 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:17.662508    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:17.662636 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662513    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-kubelet\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.662636 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662544    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.662636 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662578    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8629\" (UniqueName: \"kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629\") pod \"network-check-target-fqsbg\" (UID: \"b2eea199-27b2-47fc-8a3e-5145f9b9b54a\") " pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:17.662636 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:17.662618    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret podName:4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:18.162593789 +0000 UTC m=+2.973941141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret") pod "global-pull-secret-syncer-8tk6w" (UID: "4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:17.662973 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662670    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-kubelet\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.662973 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662688    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-socket-dir\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.662973 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662727    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/519f6723-a7eb-4bbd-82db-e6664825df3c-hosts-file\") pod \"node-resolver-fk2vd\" (UID: \"519f6723-a7eb-4bbd-82db-e6664825df3c\") " pod="openshift-dns/node-resolver-fk2vd"
Jul 27 21:45:17.662973 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662771    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-log-socket\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.662973 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662803    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-system-cni-dir\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.662973 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662829    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/135bb451-1678-49a0-b5bb-33e63a8ac0c2-host\") pod \"node-ca-75kmz\" (UID: \"135bb451-1678-49a0-b5bb-33e63a8ac0c2\") " pod="openshift-image-registry/node-ca-75kmz"
Jul 27 21:45:17.662973 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662852    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/519f6723-a7eb-4bbd-82db-e6664825df3c-hosts-file\") pod \"node-resolver-fk2vd\" (UID: \"519f6723-a7eb-4bbd-82db-e6664825df3c\") " pod="openshift-dns/node-resolver-fk2vd"
Jul 27 21:45:17.662973 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662854    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-sysctl-d\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.662973 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662893    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-log-socket\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.662973 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662892    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-sysctl-conf\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.662973 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662926    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-run\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.662973 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662952    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12060402-e3dd-4d2c-8832-11da17b477ec-tmp\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.662973 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.662976    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h8hb\" (UniqueName: \"kubernetes.io/projected/12060402-e3dd-4d2c-8832-11da17b477ec-kube-api-access-6h8hb\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663001    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-multus-cni-dir\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663045    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-etc-selinux\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663073    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-var-lib-kubelet\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663099    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663128    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-systemd-units\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663150    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-lib-modules\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663170    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-multus-conf-dir\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663195    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55q68\" (UniqueName: \"kubernetes.io/projected/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-kube-api-access-55q68\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663220    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663224    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-etc-selinux\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663247    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-run-ovn\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663288    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/67a63f7d-7ca2-4626-abcd-5138b6359eb2-konnectivity-ca\") pod \"konnectivity-agent-zl79r\" (UID: \"67a63f7d-7ca2-4626-abcd-5138b6359eb2\") " pod="kube-system/konnectivity-agent-zl79r"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663308    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-run-ovn\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663324    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-sysconfig\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663348    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-systemd\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663373    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-system-cni-dir\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.663548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663378    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-systemd-units\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663399    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-etc-kubernetes\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663424    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs\") pod \"network-metrics-daemon-hpzp8\" (UID: \"a03739ad-aa56-494b-9c4c-bd038c23af2a\") " pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663460    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-registration-dir\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663494    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-slash\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663508    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-registration-dir\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663534    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-node-log\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663560    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73750731-6722-49f8-a9d2-55df538483bf-ovnkube-config\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663611    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73750731-6722-49f8-a9d2-55df538483bf-ovn-node-metrics-cert\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663636    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73750731-6722-49f8-a9d2-55df538483bf-ovnkube-script-lib\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663660    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnqm4\" (UniqueName: \"kubernetes.io/projected/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-kube-api-access-pnqm4\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663701    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-run-systemd\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663760    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-run-openvswitch\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663789    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2z76\" (UniqueName: \"kubernetes.io/projected/135bb451-1678-49a0-b5bb-33e63a8ac0c2-kube-api-access-z2z76\") pod \"node-ca-75kmz\" (UID: \"135bb451-1678-49a0-b5bb-33e63a8ac0c2\") " pod="openshift-image-registry/node-ca-75kmz"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663818    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4hq2\" (UniqueName: \"kubernetes.io/projected/519f6723-a7eb-4bbd-82db-e6664825df3c-kube-api-access-q4hq2\") pod \"node-resolver-fk2vd\" (UID: \"519f6723-a7eb-4bbd-82db-e6664825df3c\") " pod="openshift-dns/node-resolver-fk2vd"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663846    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-etc-openvswitch\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663871    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-modprobe-d\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.664137 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663896    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-var-lib-cni-bin\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663922    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-var-lib-cni-multus\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663950    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-multus-daemon-config\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.663993    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/67a63f7d-7ca2-4626-abcd-5138b6359eb2-konnectivity-ca\") pod \"konnectivity-agent-zl79r\" (UID: \"67a63f7d-7ca2-4626-abcd-5138b6359eb2\") " pod="kube-system/konnectivity-agent-zl79r"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664041    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-dbus\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664067    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/67a63f7d-7ca2-4626-abcd-5138b6359eb2-agent-certs\") pod \"konnectivity-agent-zl79r\" (UID: \"67a63f7d-7ca2-4626-abcd-5138b6359eb2\") " pod="kube-system/konnectivity-agent-zl79r"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664096    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-run-netns\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664125    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-os-release\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664136    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-node-log\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664152    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664176    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-slash\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664179    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-var-lib-openvswitch\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664203    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73750731-6722-49f8-a9d2-55df538483bf-env-overrides\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664073    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-run-systemd\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664292    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-os-release\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664336    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/135bb451-1678-49a0-b5bb-33e63a8ac0c2-serviceca\") pod \"node-ca-75kmz\" (UID: \"135bb451-1678-49a0-b5bb-33e63a8ac0c2\") " pod="openshift-image-registry/node-ca-75kmz"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664365    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-kubelet-config\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:17.664933 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664433    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-kubelet-config\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664443    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664443    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-var-lib-openvswitch\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664487    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-run-openvswitch\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664495    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-etc-openvswitch\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664546    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73750731-6722-49f8-a9d2-55df538483bf-ovnkube-config\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664538    2549 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664687    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73750731-6722-49f8-a9d2-55df538483bf-env-overrides\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664857    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-kubernetes\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664896    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-cnibin\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664947    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-cni-binary-copy\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.664988    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-multus-socket-dir-parent\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665022    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-hostroot\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665052    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz728\" (UniqueName: \"kubernetes.io/projected/a03739ad-aa56-494b-9c4c-bd038c23af2a-kube-api-access-wz728\") pod \"network-metrics-daemon-hpzp8\" (UID: \"a03739ad-aa56-494b-9c4c-bd038c23af2a\") " pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665089    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/741209f5-9dd3-4529-9966-00a653191015-iptables-alerter-script\") pod \"iptables-alerter-nzf6x\" (UID: \"741209f5-9dd3-4529-9966-00a653191015\") " pod="openshift-network-operator/iptables-alerter-nzf6x"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665117    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-device-dir\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665121    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-dbus\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:17.665815 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665145    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-sys-fs\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665171    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-run-netns\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665178    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-device-dir\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665198    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-var-lib-kubelet\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665227    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-sys-fs\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665257    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-host\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665271    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-run-netns\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665285    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-cnibin\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665345    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-cni-binary-copy\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665383    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26ljt\" (UniqueName: \"kubernetes.io/projected/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-kube-api-access-26ljt\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665407    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-sys\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665424    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-run-multus-certs\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665441    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665458    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mttgf\" (UniqueName: \"kubernetes.io/projected/73750731-6722-49f8-a9d2-55df538483bf-kube-api-access-mttgf\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665594    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73750731-6722-49f8-a9d2-55df538483bf-ovnkube-script-lib\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.666634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.665654    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73750731-6722-49f8-a9d2-55df538483bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.669039 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.669013    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73750731-6722-49f8-a9d2-55df538483bf-ovn-node-metrics-cert\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.670386 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.670357    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/67a63f7d-7ca2-4626-abcd-5138b6359eb2-agent-certs\") pod \"konnectivity-agent-zl79r\" (UID: \"67a63f7d-7ca2-4626-abcd-5138b6359eb2\") " pod="kube-system/konnectivity-agent-zl79r"
Jul 27 21:45:17.684629 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.678455    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mttgf\" (UniqueName: \"kubernetes.io/projected/73750731-6722-49f8-a9d2-55df538483bf-kube-api-access-mttgf\") pod \"ovnkube-node-8ggqh\" (UID: \"73750731-6722-49f8-a9d2-55df538483bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.684629 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.679233    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26ljt\" (UniqueName: \"kubernetes.io/projected/b1e94b5c-5be6-4190-b949-1c6c9b212dc8-kube-api-access-26ljt\") pod \"aws-ebs-csi-driver-node-5584z\" (UID: \"b1e94b5c-5be6-4190-b949-1c6c9b212dc8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.684855 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.684677    2549 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:45:17.688073 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.688047    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4hq2\" (UniqueName: \"kubernetes.io/projected/519f6723-a7eb-4bbd-82db-e6664825df3c-kube-api-access-q4hq2\") pod \"node-resolver-fk2vd\" (UID: \"519f6723-a7eb-4bbd-82db-e6664825df3c\") " pod="openshift-dns/node-resolver-fk2vd"
Jul 27 21:45:17.700123 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.700050    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-124.ec2.internal" event={"ID":"83a6c3c959838c7e0ad7324eda9fbb06","Type":"ContainerStarted","Data":"ecce08552ab33a8499c7a6cd8a47eac3893c53d81cbc1c430f6a25c34aa83b9f"}
Jul 27 21:45:17.702600 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.702561    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal" event={"ID":"a02bc3e08ab42c32f5cc4d518b13a237","Type":"ContainerStarted","Data":"ba81073d3104af83f94a3f22acd2d7d7150b49b264e39e2a0d41cf3e3340f462"}
Jul 27 21:45:17.766245 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766189    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-os-release\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.766245 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766233    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-os-release\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.766516 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766279    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/135bb451-1678-49a0-b5bb-33e63a8ac0c2-serviceca\") pod \"node-ca-75kmz\" (UID: \"135bb451-1678-49a0-b5bb-33e63a8ac0c2\") " pod="openshift-image-registry/node-ca-75kmz"
Jul 27 21:45:17.766516 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766306    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-kubernetes\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.766516 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766331    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-cnibin\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.766516 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766356    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-cni-binary-copy\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.766516 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766392    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-multus-socket-dir-parent\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.766516 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766414    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-hostroot\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.766516 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766440    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz728\" (UniqueName: \"kubernetes.io/projected/a03739ad-aa56-494b-9c4c-bd038c23af2a-kube-api-access-wz728\") pod \"network-metrics-daemon-hpzp8\" (UID: \"a03739ad-aa56-494b-9c4c-bd038c23af2a\") " pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:17.766516 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766465    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/741209f5-9dd3-4529-9966-00a653191015-iptables-alerter-script\") pod \"iptables-alerter-nzf6x\" (UID: \"741209f5-9dd3-4529-9966-00a653191015\") " pod="openshift-network-operator/iptables-alerter-nzf6x"
Jul 27 21:45:17.766516 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766495    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-var-lib-kubelet\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.766516 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766519    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-host\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766545    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-cnibin\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766573    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-cni-binary-copy\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766598    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-sys\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766623    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-run-multus-certs\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766666    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/12060402-e3dd-4d2c-8832-11da17b477ec-etc-tuned\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766692    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-run-k8s-cni-cncf-io\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766719    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/741209f5-9dd3-4529-9966-00a653191015-host-slash\") pod \"iptables-alerter-nzf6x\" (UID: \"741209f5-9dd3-4529-9966-00a653191015\") " pod="openshift-network-operator/iptables-alerter-nzf6x"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766744    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p52wd\" (UniqueName: \"kubernetes.io/projected/741209f5-9dd3-4529-9966-00a653191015-kube-api-access-p52wd\") pod \"iptables-alerter-nzf6x\" (UID: \"741209f5-9dd3-4529-9966-00a653191015\") " pod="openshift-network-operator/iptables-alerter-nzf6x"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766775    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766793    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/135bb451-1678-49a0-b5bb-33e63a8ac0c2-serviceca\") pod \"node-ca-75kmz\" (UID: \"135bb451-1678-49a0-b5bb-33e63a8ac0c2\") " pod="openshift-image-registry/node-ca-75kmz"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766803    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8629\" (UniqueName: \"kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629\") pod \"network-check-target-fqsbg\" (UID: \"b2eea199-27b2-47fc-8a3e-5145f9b9b54a\") " pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766831    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-system-cni-dir\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766855    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/135bb451-1678-49a0-b5bb-33e63a8ac0c2-host\") pod \"node-ca-75kmz\" (UID: \"135bb451-1678-49a0-b5bb-33e63a8ac0c2\") " pod="openshift-image-registry/node-ca-75kmz"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766880    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-os-release\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766912    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-sysctl-d\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766950    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-kubernetes\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.767124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.767002    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-cnibin\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.768013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766331    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-os-release\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.768013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.767013    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/135bb451-1678-49a0-b5bb-33e63a8ac0c2-host\") pod \"node-ca-75kmz\" (UID: \"135bb451-1678-49a0-b5bb-33e63a8ac0c2\") " pod="openshift-image-registry/node-ca-75kmz"
Jul 27 21:45:17.768013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.767063    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-multus-socket-dir-parent\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.768013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.767103    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-hostroot\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.768013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.767137    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-sysctl-d\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.768013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.767219    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-run-multus-certs\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.768013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.767642    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-cni-binary-copy\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.768013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.767674    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-cnibin\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.768013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.767681    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/741209f5-9dd3-4529-9966-00a653191015-iptables-alerter-script\") pod \"iptables-alerter-nzf6x\" (UID: \"741209f5-9dd3-4529-9966-00a653191015\") " pod="openshift-network-operator/iptables-alerter-nzf6x"
Jul 27 21:45:17.768013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.767720    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-sys\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.768013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.767767    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-host\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.768013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.767778    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-var-lib-kubelet\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.768013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.767826    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-run-k8s-cni-cncf-io\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.768013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.767873    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/741209f5-9dd3-4529-9966-00a653191015-host-slash\") pod \"iptables-alerter-nzf6x\" (UID: \"741209f5-9dd3-4529-9966-00a653191015\") " pod="openshift-network-operator/iptables-alerter-nzf6x"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768039    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-system-cni-dir\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768130    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768136    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-cni-binary-copy\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.766940    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-sysctl-conf\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768203    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-run\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768226    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12060402-e3dd-4d2c-8832-11da17b477ec-tmp\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768251    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h8hb\" (UniqueName: \"kubernetes.io/projected/12060402-e3dd-4d2c-8832-11da17b477ec-kube-api-access-6h8hb\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768277    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-multus-cni-dir\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768297    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-sysctl-conf\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768325    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-var-lib-kubelet\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768354    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768380    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-lib-modules\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768377    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-run\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768402    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-multus-conf-dir\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768427    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55q68\" (UniqueName: \"kubernetes.io/projected/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-kube-api-access-55q68\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768456    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.768718 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768485    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-sysconfig\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768508    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-systemd\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768533    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-system-cni-dir\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768557    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-etc-kubernetes\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768583    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs\") pod \"network-metrics-daemon-hpzp8\" (UID: \"a03739ad-aa56-494b-9c4c-bd038c23af2a\") " pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768595    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-multus-cni-dir\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768616    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnqm4\" (UniqueName: \"kubernetes.io/projected/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-kube-api-access-pnqm4\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768644    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2z76\" (UniqueName: \"kubernetes.io/projected/135bb451-1678-49a0-b5bb-33e63a8ac0c2-kube-api-access-z2z76\") pod \"node-ca-75kmz\" (UID: \"135bb451-1678-49a0-b5bb-33e63a8ac0c2\") " pod="openshift-image-registry/node-ca-75kmz"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768673    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-modprobe-d\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768699    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-system-cni-dir\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768758    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-systemd\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768780    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-lib-modules\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:17.768787    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768836    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-multus-conf-dir\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768844    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-var-lib-kubelet\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768839    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-sysconfig\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768700    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-var-lib-cni-bin\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768796    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-var-lib-cni-bin\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.769577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768795    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/12060402-e3dd-4d2c-8832-11da17b477ec-etc-modprobe-d\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.770482 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768885    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-etc-kubernetes\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.770482 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:17.768894    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs podName:a03739ad-aa56-494b-9c4c-bd038c23af2a nodeName:}" failed. No retries permitted until 2026-07-27 21:45:18.268871769 +0000 UTC m=+3.080219123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs") pod "network-metrics-daemon-hpzp8" (UID: "a03739ad-aa56-494b-9c4c-bd038c23af2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:17.770482 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768940    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-var-lib-cni-multus\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.770482 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.768973    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-multus-daemon-config\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.770482 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.769004    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-run-netns\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.770482 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.769059    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-var-lib-cni-multus\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.770482 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.769076    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-host-run-netns\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.770482 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.769292    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.770482 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.769460    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-multus-daemon-config\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.770482 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.769875    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.770482 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.770130    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/12060402-e3dd-4d2c-8832-11da17b477ec-etc-tuned\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.771139 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.771121    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12060402-e3dd-4d2c-8832-11da17b477ec-tmp\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.777211 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:17.777142    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:45:17.777211 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:17.777171    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:45:17.777211 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:17.777186    2549 projected.go:196] Error preparing data for projected volume kube-api-access-d8629 for pod openshift-network-diagnostics/network-check-target-fqsbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:17.777507 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:17.777299    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629 podName:b2eea199-27b2-47fc-8a3e-5145f9b9b54a nodeName:}" failed. No retries permitted until 2026-07-27 21:45:18.277275849 +0000 UTC m=+3.088623204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d8629" (UniqueName: "kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629") pod "network-check-target-fqsbg" (UID: "b2eea199-27b2-47fc-8a3e-5145f9b9b54a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:17.778910 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.778881    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p52wd\" (UniqueName: \"kubernetes.io/projected/741209f5-9dd3-4529-9966-00a653191015-kube-api-access-p52wd\") pod \"iptables-alerter-nzf6x\" (UID: \"741209f5-9dd3-4529-9966-00a653191015\") " pod="openshift-network-operator/iptables-alerter-nzf6x"
Jul 27 21:45:17.779534 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.779507    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnqm4\" (UniqueName: \"kubernetes.io/projected/b63fbb6e-0ff3-4e2c-96f8-2717feec7816-kube-api-access-pnqm4\") pod \"multus-additional-cni-plugins-b9s8g\" (UID: \"b63fbb6e-0ff3-4e2c-96f8-2717feec7816\") " pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.779642 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.779546    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2z76\" (UniqueName: \"kubernetes.io/projected/135bb451-1678-49a0-b5bb-33e63a8ac0c2-kube-api-access-z2z76\") pod \"node-ca-75kmz\" (UID: \"135bb451-1678-49a0-b5bb-33e63a8ac0c2\") " pod="openshift-image-registry/node-ca-75kmz"
Jul 27 21:45:17.780582 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.780558    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz728\" (UniqueName: \"kubernetes.io/projected/a03739ad-aa56-494b-9c4c-bd038c23af2a-kube-api-access-wz728\") pod \"network-metrics-daemon-hpzp8\" (UID: \"a03739ad-aa56-494b-9c4c-bd038c23af2a\") " pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:17.780808 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.780783    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h8hb\" (UniqueName: \"kubernetes.io/projected/12060402-e3dd-4d2c-8832-11da17b477ec-kube-api-access-6h8hb\") pod \"tuned-fwl7s\" (UID: \"12060402-e3dd-4d2c-8832-11da17b477ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.781845 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.781814    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55q68\" (UniqueName: \"kubernetes.io/projected/caa2e52d-0fe9-4323-8d4e-da36f5c099dd-kube-api-access-55q68\") pod \"multus-hk9dv\" (UID: \"caa2e52d-0fe9-4323-8d4e-da36f5c099dd\") " pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.858779 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.858737    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fk2vd"
Jul 27 21:45:17.868655 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.868624    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z"
Jul 27 21:45:17.880496 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.880466    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:17.885769 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.885746    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zl79r"
Jul 27 21:45:17.895726 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.895698    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fwl7s"
Jul 27 21:45:17.911069 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.910841    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hk9dv"
Jul 27 21:45:17.917932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.917903    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b9s8g"
Jul 27 21:45:17.927033 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.927011    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-75kmz"
Jul 27 21:45:17.934997 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:17.934970    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nzf6x"
Jul 27 21:45:18.172737 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:18.172699    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:18.172914 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:18.172846    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:18.172991 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:18.172913    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret podName:4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:19.172895298 +0000 UTC m=+3.984242651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret") pod "global-pull-secret-syncer-8tk6w" (UID: "4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:18.273989 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:18.273943    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs\") pod \"network-metrics-daemon-hpzp8\" (UID: \"a03739ad-aa56-494b-9c4c-bd038c23af2a\") " pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:18.274176 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:18.274133    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:18.274270 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:18.274255    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs podName:a03739ad-aa56-494b-9c4c-bd038c23af2a nodeName:}" failed. No retries permitted until 2026-07-27 21:45:19.27419729 +0000 UTC m=+4.085544651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs") pod "network-metrics-daemon-hpzp8" (UID: "a03739ad-aa56-494b-9c4c-bd038c23af2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:18.374462 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:18.374413    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8629\" (UniqueName: \"kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629\") pod \"network-check-target-fqsbg\" (UID: \"b2eea199-27b2-47fc-8a3e-5145f9b9b54a\") " pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:18.374668 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:18.374650    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:45:18.374752 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:18.374683    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:45:18.374752 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:18.374700    2549 projected.go:196] Error preparing data for projected volume kube-api-access-d8629 for pod openshift-network-diagnostics/network-check-target-fqsbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:18.374858 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:18.374765    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629 podName:b2eea199-27b2-47fc-8a3e-5145f9b9b54a nodeName:}" failed. No retries permitted until 2026-07-27 21:45:19.374741711 +0000 UTC m=+4.186089121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8629" (UniqueName: "kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629") pod "network-check-target-fqsbg" (UID: "b2eea199-27b2-47fc-8a3e-5145f9b9b54a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:18.605407 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:18.605358    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 21:40:18 +0000 UTC" deadline="2027-12-31 20:41:03.831619485 +0000 UTC"
Jul 27 21:45:18.605407 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:18.605399    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12526h55m45.226225102s"
Jul 27 21:45:18.854172 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:18.854112    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12060402_e3dd_4d2c_8832_11da17b477ec.slice/crio-c4346abcfa5c69143bce31346e62178a53f31c05e732f7aa3352d1557131176e WatchSource:0}: Error finding container c4346abcfa5c69143bce31346e62178a53f31c05e732f7aa3352d1557131176e: Status 404 returned error can't find the container with id c4346abcfa5c69143bce31346e62178a53f31c05e732f7aa3352d1557131176e
Jul 27 21:45:18.887805 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:18.887753    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741209f5_9dd3_4529_9966_00a653191015.slice/crio-ecc928e2400dc363f3567b6c4c30118e2a09713872234a88f5771cb5ed20d8d5 WatchSource:0}: Error finding container ecc928e2400dc363f3567b6c4c30118e2a09713872234a88f5771cb5ed20d8d5: Status 404 returned error can't find the container with id ecc928e2400dc363f3567b6c4c30118e2a09713872234a88f5771cb5ed20d8d5
Jul 27 21:45:19.179971 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.179770    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:19.180144 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:19.179946    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:19.180144 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:19.180050    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret podName:4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:21.180034814 +0000 UTC m=+5.991382168 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret") pod "global-pull-secret-syncer-8tk6w" (UID: "4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:19.281193 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.281147    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs\") pod \"network-metrics-daemon-hpzp8\" (UID: \"a03739ad-aa56-494b-9c4c-bd038c23af2a\") " pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:19.281409 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:19.281324    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:19.281409 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:19.281409    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs podName:a03739ad-aa56-494b-9c4c-bd038c23af2a nodeName:}" failed. No retries permitted until 2026-07-27 21:45:21.281388898 +0000 UTC m=+6.092736264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs") pod "network-metrics-daemon-hpzp8" (UID: "a03739ad-aa56-494b-9c4c-bd038c23af2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:19.382210 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.382088    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8629\" (UniqueName: \"kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629\") pod \"network-check-target-fqsbg\" (UID: \"b2eea199-27b2-47fc-8a3e-5145f9b9b54a\") " pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:19.382403 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:19.382271    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:45:19.382403 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:19.382355    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:45:19.382403 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:19.382368    2549 projected.go:196] Error preparing data for projected volume kube-api-access-d8629 for pod openshift-network-diagnostics/network-check-target-fqsbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:19.382537 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:19.382433    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629 podName:b2eea199-27b2-47fc-8a3e-5145f9b9b54a nodeName:}" failed. No retries permitted until 2026-07-27 21:45:21.382417238 +0000 UTC m=+6.193764594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8629" (UniqueName: "kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629") pod "network-check-target-fqsbg" (UID: "b2eea199-27b2-47fc-8a3e-5145f9b9b54a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:19.605910 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.605839    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 21:40:18 +0000 UTC" deadline="2028-01-08 02:20:58.137704583 +0000 UTC"
Jul 27 21:45:19.605910 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.605908    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12700h35m38.531800437s"
Jul 27 21:45:19.696943 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.696857    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:19.697149 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:19.696988    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:19.697149 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.697030    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:19.697149 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:19.697120    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:19.697354 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.697177    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:19.697354 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:19.697241    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:19.716253 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.716214    2549 generic.go:356] "Generic (PLEG): container finished" podID="a02bc3e08ab42c32f5cc4d518b13a237" containerID="2f161007901a0e40f1b9a9b5c58e44d927c7ca685328876709ce0113665988bb" exitCode=0
Jul 27 21:45:19.716458 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.716330    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal" event={"ID":"a02bc3e08ab42c32f5cc4d518b13a237","Type":"ContainerDied","Data":"2f161007901a0e40f1b9a9b5c58e44d927c7ca685328876709ce0113665988bb"}
Jul 27 21:45:19.722037 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.722002    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh" event={"ID":"73750731-6722-49f8-a9d2-55df538483bf","Type":"ContainerStarted","Data":"4e7b6a8edeb22c6ecbc5d23aa24fbe6ba2a8fabf6e6a04cb01b49e8864c93c3d"}
Jul 27 21:45:19.726772 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.726740    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-75kmz" event={"ID":"135bb451-1678-49a0-b5bb-33e63a8ac0c2","Type":"ContainerStarted","Data":"46443afdd42b78d7f0712aa6b689fbb58f506c2eabd73a88830ca3d55a108c66"}
Jul 27 21:45:19.736866 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.736835    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9s8g" event={"ID":"b63fbb6e-0ff3-4e2c-96f8-2717feec7816","Type":"ContainerStarted","Data":"f86200b597009d0a433b0b1a5b2c6e12ae4040672be482992bf6537546682107"}
Jul 27 21:45:19.747338 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.747284    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fk2vd" event={"ID":"519f6723-a7eb-4bbd-82db-e6664825df3c","Type":"ContainerStarted","Data":"d185bd6607d8742f4a4291ce149fd775669f789e8e49a35932bcd44bc336aa90"}
Jul 27 21:45:19.753051 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.752335    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-124.ec2.internal" event={"ID":"83a6c3c959838c7e0ad7324eda9fbb06","Type":"ContainerStarted","Data":"b80074166b49c499b7d5ebbd2758371ee29119ad0342796d5e197883210018f2"}
Jul 27 21:45:19.757954 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.756960    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z" event={"ID":"b1e94b5c-5be6-4190-b949-1c6c9b212dc8","Type":"ContainerStarted","Data":"a3cb4d0ee3dc06f75fd499afbbd132c582ec683b3e5a56c373dacaa531b561af"}
Jul 27 21:45:19.761932 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.761861    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hk9dv" event={"ID":"caa2e52d-0fe9-4323-8d4e-da36f5c099dd","Type":"ContainerStarted","Data":"ae53297b25f06d8b556c6137455722dc0341ad962116297f955f49e4bff93eaa"}
Jul 27 21:45:19.771540 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.770201    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-124.ec2.internal" podStartSLOduration=2.770181807 podStartE2EDuration="2.770181807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:19.770168059 +0000 UTC m=+4.581515425" watchObservedRunningTime="2026-07-27 21:45:19.770181807 +0000 UTC m=+4.581529183"
Jul 27 21:45:19.771916 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.771888    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zl79r" event={"ID":"67a63f7d-7ca2-4626-abcd-5138b6359eb2","Type":"ContainerStarted","Data":"ca20b0c2922a20ad124cb34a443209770be7cb99ac7d1bf3a18d87ff79d52276"}
Jul 27 21:45:19.775259 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.774436    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nzf6x" event={"ID":"741209f5-9dd3-4529-9966-00a653191015","Type":"ContainerStarted","Data":"ecc928e2400dc363f3567b6c4c30118e2a09713872234a88f5771cb5ed20d8d5"}
Jul 27 21:45:19.792820 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:19.792776    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fwl7s" event={"ID":"12060402-e3dd-4d2c-8832-11da17b477ec","Type":"ContainerStarted","Data":"c4346abcfa5c69143bce31346e62178a53f31c05e732f7aa3352d1557131176e"}
Jul 27 21:45:20.830465 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:20.830420    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal" event={"ID":"a02bc3e08ab42c32f5cc4d518b13a237","Type":"ContainerStarted","Data":"fc5ab5c192a487b4e65cd1c4db6b17bb01f4bceff3e29ae7f053edb153e13d94"}
Jul 27 21:45:20.847565 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:20.846946    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-124.ec2.internal" podStartSLOduration=3.8469208950000002 podStartE2EDuration="3.846920895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:20.8463898 +0000 UTC m=+5.657737177" watchObservedRunningTime="2026-07-27 21:45:20.846920895 +0000 UTC m=+5.658268272"
Jul 27 21:45:21.199198 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:21.199160    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:21.199412 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:21.199374    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:21.199489 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:21.199441    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret podName:4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:25.199419941 +0000 UTC m=+10.010767300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret") pod "global-pull-secret-syncer-8tk6w" (UID: "4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:21.299913 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:21.299869    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs\") pod \"network-metrics-daemon-hpzp8\" (UID: \"a03739ad-aa56-494b-9c4c-bd038c23af2a\") " pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:21.300098 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:21.300073    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:21.300183 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:21.300143    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs podName:a03739ad-aa56-494b-9c4c-bd038c23af2a nodeName:}" failed. No retries permitted until 2026-07-27 21:45:25.300122055 +0000 UTC m=+10.111469423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs") pod "network-metrics-daemon-hpzp8" (UID: "a03739ad-aa56-494b-9c4c-bd038c23af2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:21.402006 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:21.401315    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8629\" (UniqueName: \"kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629\") pod \"network-check-target-fqsbg\" (UID: \"b2eea199-27b2-47fc-8a3e-5145f9b9b54a\") " pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:21.402006 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:21.401525    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:45:21.402006 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:21.401550    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:45:21.402006 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:21.401563    2549 projected.go:196] Error preparing data for projected volume kube-api-access-d8629 for pod openshift-network-diagnostics/network-check-target-fqsbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:21.402006 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:21.401636    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629 podName:b2eea199-27b2-47fc-8a3e-5145f9b9b54a nodeName:}" failed. No retries permitted until 2026-07-27 21:45:25.401614057 +0000 UTC m=+10.212961417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8629" (UniqueName: "kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629") pod "network-check-target-fqsbg" (UID: "b2eea199-27b2-47fc-8a3e-5145f9b9b54a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:21.695432 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:21.695398    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:21.695651 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:21.695550    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:21.696008 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:21.695984    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:21.696138 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:21.696114    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:21.696454 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:21.696321    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:21.696454 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:21.696416    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:23.696165 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:23.696135    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:23.696581 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:23.696315    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:23.697202 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:23.696937    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:23.697202 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:23.697044    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:23.697202 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:23.697088    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:23.697202 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:23.697156    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:25.236000 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:25.235958    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:25.236523 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:25.236191    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:25.236523 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:25.236281    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret podName:4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:33.23624002 +0000 UTC m=+18.047587398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret") pod "global-pull-secret-syncer-8tk6w" (UID: "4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:25.336949 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:25.336904    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs\") pod \"network-metrics-daemon-hpzp8\" (UID: \"a03739ad-aa56-494b-9c4c-bd038c23af2a\") " pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:25.337145 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:25.337091    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:25.337209 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:25.337162    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs podName:a03739ad-aa56-494b-9c4c-bd038c23af2a nodeName:}" failed. No retries permitted until 2026-07-27 21:45:33.337140216 +0000 UTC m=+18.148487568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs") pod "network-metrics-daemon-hpzp8" (UID: "a03739ad-aa56-494b-9c4c-bd038c23af2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:25.438187 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:25.438144    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8629\" (UniqueName: \"kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629\") pod \"network-check-target-fqsbg\" (UID: \"b2eea199-27b2-47fc-8a3e-5145f9b9b54a\") " pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:25.438513 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:25.438321    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:45:25.438513 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:25.438337    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:45:25.438513 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:25.438347    2549 projected.go:196] Error preparing data for projected volume kube-api-access-d8629 for pod openshift-network-diagnostics/network-check-target-fqsbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:25.438513 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:25.438408    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629 podName:b2eea199-27b2-47fc-8a3e-5145f9b9b54a nodeName:}" failed. No retries permitted until 2026-07-27 21:45:33.438387173 +0000 UTC m=+18.249734544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8629" (UniqueName: "kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629") pod "network-check-target-fqsbg" (UID: "b2eea199-27b2-47fc-8a3e-5145f9b9b54a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:25.695568 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:25.695532    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:25.695759 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:25.695654    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:25.696156 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:25.696138    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:25.696258 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:25.696146    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:25.696258 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:25.696251    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:25.696457 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:25.696383    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:27.694568 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:27.694528    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:27.695073 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:27.694538    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:27.695073 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:27.694677    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:27.695073 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:27.694678    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:27.695073 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:27.694794    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:27.695073 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:27.694941    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:29.694603 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:29.694565    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:29.695037 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:29.694636    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:29.695037 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:29.694646    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:29.695037 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:29.694734    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:29.695037 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:29.694808    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:29.695037 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:29.694861    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:31.694185 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:31.694148    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:31.694641 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:31.694149    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:31.694641 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:31.694285    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:31.694641 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:31.694170    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:31.694641 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:31.694341    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:31.694641 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:31.694456    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:33.302581 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:33.302536    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:33.303145 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:33.302700    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:33.303145 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:33.302768    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret podName:4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:49.302750602 +0000 UTC m=+34.114097953 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret") pod "global-pull-secret-syncer-8tk6w" (UID: "4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:33.403306 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:33.403272    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs\") pod \"network-metrics-daemon-hpzp8\" (UID: \"a03739ad-aa56-494b-9c4c-bd038c23af2a\") " pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:33.403479 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:33.403411    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:33.403479 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:33.403468    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs podName:a03739ad-aa56-494b-9c4c-bd038c23af2a nodeName:}" failed. No retries permitted until 2026-07-27 21:45:49.403455016 +0000 UTC m=+34.214802367 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs") pod "network-metrics-daemon-hpzp8" (UID: "a03739ad-aa56-494b-9c4c-bd038c23af2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:33.504701 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:33.504657    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8629\" (UniqueName: \"kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629\") pod \"network-check-target-fqsbg\" (UID: \"b2eea199-27b2-47fc-8a3e-5145f9b9b54a\") " pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:33.504878 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:33.504862    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:45:33.504937 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:33.504887    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:45:33.504937 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:33.504900    2549 projected.go:196] Error preparing data for projected volume kube-api-access-d8629 for pod openshift-network-diagnostics/network-check-target-fqsbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:33.505041 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:33.504965    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629 podName:b2eea199-27b2-47fc-8a3e-5145f9b9b54a nodeName:}" failed. No retries permitted until 2026-07-27 21:45:49.504946746 +0000 UTC m=+34.316294100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8629" (UniqueName: "kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629") pod "network-check-target-fqsbg" (UID: "b2eea199-27b2-47fc-8a3e-5145f9b9b54a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:33.694521 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:33.694481    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:33.694521 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:33.694522    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:33.694764 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:33.694481    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:33.694764 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:33.694605    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:33.694764 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:33.694705    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:33.694896 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:33.694770    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:35.694811 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:35.694777    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:35.695363 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:35.694872    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:35.695363 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:35.694891    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:35.695363 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:35.694960    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:35.695363 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:35.694986    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:35.695363 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:35.695096    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:37.694665 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:37.694631    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:37.695185 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:37.694694    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:37.695185 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:37.694707    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:37.695185 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:37.694829    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:37.695185 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:37.694908    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:37.695185 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:37.695001    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:38.867830 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.867434    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z" event={"ID":"b1e94b5c-5be6-4190-b949-1c6c9b212dc8","Type":"ContainerStarted","Data":"779098c12d0ba42bdcc4b8634c4d26e0dc17e941969c5caab1684908086fa2ef"}
Jul 27 21:45:38.869375 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.869345    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hk9dv" event={"ID":"caa2e52d-0fe9-4323-8d4e-da36f5c099dd","Type":"ContainerStarted","Data":"505016df8ab938ae894cb5a180de0af24ed08da55b853b6a7e6d9cc8083b24bd"}
Jul 27 21:45:38.871690 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.871378    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zl79r" event={"ID":"67a63f7d-7ca2-4626-abcd-5138b6359eb2","Type":"ContainerStarted","Data":"8d6f887d0a7b4b3f2ad79a401678366f86a7e6bb4131a11c30f8af7548581499"}
Jul 27 21:45:38.873198 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.873172    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fwl7s" event={"ID":"12060402-e3dd-4d2c-8832-11da17b477ec","Type":"ContainerStarted","Data":"1b9764bbcf01e42e660aea712b0bbd4c2cfb72d1ddf0189d69a12d48c276f49b"}
Jul 27 21:45:38.876690 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.876666    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh" event={"ID":"73750731-6722-49f8-a9d2-55df538483bf","Type":"ContainerStarted","Data":"7ce7988ce7c90f22840d756bd02406a83f762d96366a5a7d6861eef473bb1d3b"}
Jul 27 21:45:38.876781 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.876738    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh" event={"ID":"73750731-6722-49f8-a9d2-55df538483bf","Type":"ContainerStarted","Data":"660fbd1f41b267a9ef4238102d3f61a47eff0e31ebec672aa4d15390ba363ee3"}
Jul 27 21:45:38.876824 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.876787    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh" event={"ID":"73750731-6722-49f8-a9d2-55df538483bf","Type":"ContainerStarted","Data":"9ee07303cbb800d5af8755a7f8f2a80c73233ae8d04e1df6a70bc3eb50446fc5"}
Jul 27 21:45:38.876867 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.876841    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh" event={"ID":"73750731-6722-49f8-a9d2-55df538483bf","Type":"ContainerStarted","Data":"97f7497a944837414641677ac7b4443acede70e873b6f8f3d7be0a8b9c39d14c"}
Jul 27 21:45:38.876901 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.876887    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh" event={"ID":"73750731-6722-49f8-a9d2-55df538483bf","Type":"ContainerStarted","Data":"ae5aed64e457d92aa50edec502934f1f9fbbd728983f1c21f1d440b8a9da7322"}
Jul 27 21:45:38.876965 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.876933    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh" event={"ID":"73750731-6722-49f8-a9d2-55df538483bf","Type":"ContainerStarted","Data":"025d9203f1bec60a67ce2dba37e8e12201f3b1585dadf56a65e60e3d468bc709"}
Jul 27 21:45:38.878174 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.878149    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-75kmz" event={"ID":"135bb451-1678-49a0-b5bb-33e63a8ac0c2","Type":"ContainerStarted","Data":"306167faa56f7412aa3bd682849ea9245be3bb62377a507685c38af15df1b0f5"}
Jul 27 21:45:38.879902 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.879846    2549 generic.go:356] "Generic (PLEG): container finished" podID="b63fbb6e-0ff3-4e2c-96f8-2717feec7816" containerID="9210d839d71258fe2c9df05e1edc1d02de2efb8aa62dd90d0cb0a0630b6dc2d7" exitCode=0
Jul 27 21:45:38.880012 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.879910    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9s8g" event={"ID":"b63fbb6e-0ff3-4e2c-96f8-2717feec7816","Type":"ContainerDied","Data":"9210d839d71258fe2c9df05e1edc1d02de2efb8aa62dd90d0cb0a0630b6dc2d7"}
Jul 27 21:45:38.882161 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.882131    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fk2vd" event={"ID":"519f6723-a7eb-4bbd-82db-e6664825df3c","Type":"ContainerStarted","Data":"4fa3a85e72d09af55b7db2c9bd99cfe4193f0861bdd5ce89a02bed094e8ba01d"}
Jul 27 21:45:38.892340 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.892266    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hk9dv" podStartSLOduration=3.588343282 podStartE2EDuration="22.892249064s" totalImagesPullingTime="19.303905782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:38.891180494 +0000 UTC m=+23.702527867" watchObservedRunningTime="2026-07-27 21:45:38.892249064 +0000 UTC m=+23.703596437"
Jul 27 21:45:38.906664 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.906592    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fwl7s" podStartSLOduration=3.658434954 podStartE2EDuration="22.906579344s" totalImagesPullingTime="19.24814439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:38.906415214 +0000 UTC m=+23.717762586" watchObservedRunningTime="2026-07-27 21:45:38.906579344 +0000 UTC m=+23.717926716"
Jul 27 21:45:38.972886 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:38.972838    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-75kmz" podStartSLOduration=3.740274048 podStartE2EDuration="22.972823407s" totalImagesPullingTime="19.232549359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:38.971792027 +0000 UTC m=+23.783139400" watchObservedRunningTime="2026-07-27 21:45:38.972823407 +0000 UTC m=+23.784170792"
Jul 27 21:45:39.004294 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:39.004217    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fk2vd" podStartSLOduration=3.769578455 podStartE2EDuration="23.004195862s" totalImagesPullingTime="19.234617407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:39.003780908 +0000 UTC m=+23.815128277" watchObservedRunningTime="2026-07-27 21:45:39.004195862 +0000 UTC m=+23.815543234"
Jul 27 21:45:39.021240 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:39.021186    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zl79r" podStartSLOduration=7.9624862400000005 podStartE2EDuration="23.021172096s" totalImagesPullingTime="15.058685856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:39.020825053 +0000 UTC m=+23.832172426" watchObservedRunningTime="2026-07-27 21:45:39.021172096 +0000 UTC m=+23.832519467"
Jul 27 21:45:39.634111 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:39.634085    2549 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 21:45:39.695986 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:39.695957    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:39.696168 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:39.696065    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:39.696417 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:39.696403    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:39.696494 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:39.696479    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:39.696544 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:39.696532    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:39.696618 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:39.696592    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:39.888435 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:39.888360    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nzf6x" event={"ID":"741209f5-9dd3-4529-9966-00a653191015","Type":"ContainerStarted","Data":"6c739f8b93de4e4fae17c9e6c1d295118bcf3b40ba2578abb98bef23d1f34971"}
Jul 27 21:45:39.890674 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:39.890644    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z" event={"ID":"b1e94b5c-5be6-4190-b949-1c6c9b212dc8","Type":"ContainerStarted","Data":"03f38dc791fc0efa6cdb850c879dde594166a47c71b283f635ebb9d24d53b3c6"}
Jul 27 21:45:40.622455 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:40.622358    2549 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T21:45:39.634108585Z","UUID":"bd499add-9ebf-4b91-a237-a50aa8b8a784","Handler":null,"Name":"","Endpoint":""}
Jul 27 21:45:40.624120 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:40.624092    2549 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 21:45:40.624277 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:40.624136    2549 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 21:45:41.694934 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:41.694897    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:41.695449 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:41.694933    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:41.695449 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:41.695017    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:41.695449 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:41.695030    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:41.695449 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:41.695123    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:41.695449 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:41.695240    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:41.898192 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:41.898152    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z" event={"ID":"b1e94b5c-5be6-4190-b949-1c6c9b212dc8","Type":"ContainerStarted","Data":"a5b109a5f20a28ca9c32a366839c12b6dafdf39743f0cb5dee616b40422e02b8"}
Jul 27 21:45:41.901830 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:41.901801    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh" event={"ID":"73750731-6722-49f8-a9d2-55df538483bf","Type":"ContainerStarted","Data":"8e28004b61bfe86e8a2a7320f52cb5f34e4ccae7c4d8984074357f5d8a6d6822"}
Jul 27 21:45:41.917241 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:41.917177    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nzf6x" podStartSLOduration=5.68360294 podStartE2EDuration="24.917161521s" totalImagesPullingTime="19.233558581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:39.905910488 +0000 UTC m=+24.717257861" watchObservedRunningTime="2026-07-27 21:45:41.917161521 +0000 UTC m=+26.728508893"
Jul 27 21:45:41.917391 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:41.917295    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5584z" podStartSLOduration=4.248919823 podStartE2EDuration="25.917288736s" totalImagesPullingTime="21.668368913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:41.916739981 +0000 UTC m=+26.728087345" watchObservedRunningTime="2026-07-27 21:45:41.917288736 +0000 UTC m=+26.728636107"
Jul 27 21:45:42.492499 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:42.492462    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zl79r"
Jul 27 21:45:42.736371 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:42.736330    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zl79r"
Jul 27 21:45:42.737037 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:42.737015    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zl79r"
Jul 27 21:45:42.904384 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:42.904354    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zl79r"
Jul 27 21:45:43.694956 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:43.694754    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:43.695123 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:43.694769    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:43.695123 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:43.695031    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:43.695123 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:43.694771    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:43.695329 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:43.695115    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:43.695329 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:43.695201    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:43.909132 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:43.909094    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh" event={"ID":"73750731-6722-49f8-a9d2-55df538483bf","Type":"ContainerStarted","Data":"92889985cd9e9778c3e480c0a6cb2f57f0e6672214e61fe23a61900b3ee65ec2"}
Jul 27 21:45:43.909978 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:43.909409    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:43.910973 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:43.910948    2549 generic.go:356] "Generic (PLEG): container finished" podID="b63fbb6e-0ff3-4e2c-96f8-2717feec7816" containerID="b01071745cb1bc5f885a9f0dbaca43e87de91d377aa0e5f64c1941f25400c4dd" exitCode=0
Jul 27 21:45:43.911102 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:43.911021    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9s8g" event={"ID":"b63fbb6e-0ff3-4e2c-96f8-2717feec7816","Type":"ContainerDied","Data":"b01071745cb1bc5f885a9f0dbaca43e87de91d377aa0e5f64c1941f25400c4dd"}
Jul 27 21:45:43.926603 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:43.926583    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:43.940214 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:43.940173    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh" podStartSLOduration=8.640791068 podStartE2EDuration="27.940159594s" totalImagesPullingTime="19.299368526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:43.938170273 +0000 UTC m=+28.749517692" watchObservedRunningTime="2026-07-27 21:45:43.940159594 +0000 UTC m=+28.751506964"
Jul 27 21:45:44.914338 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:44.914165    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:44.914769 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:44.914433    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:44.930399 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:44.930370    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:45:45.265675 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:45.265642    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fqsbg"]
Jul 27 21:45:45.265860 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:45.265756    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:45.265860 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:45.265843    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:45.268279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:45.268256    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hpzp8"]
Jul 27 21:45:45.268430 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:45.268368    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:45.268482 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:45.268457    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:45.268812 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:45.268790    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8tk6w"]
Jul 27 21:45:45.268899 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:45.268886    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:45.269014 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:45.268986    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:45.917320 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:45.917287    2549 generic.go:356] "Generic (PLEG): container finished" podID="b63fbb6e-0ff3-4e2c-96f8-2717feec7816" containerID="f810803cfc5e84e1d0020e8295af032d81856afd51e74ff2d8a33bea1daaa251" exitCode=0
Jul 27 21:45:45.917748 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:45.917381    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9s8g" event={"ID":"b63fbb6e-0ff3-4e2c-96f8-2717feec7816","Type":"ContainerDied","Data":"f810803cfc5e84e1d0020e8295af032d81856afd51e74ff2d8a33bea1daaa251"}
Jul 27 21:45:46.694076 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:46.694012    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:46.694076 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:46.694051    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:46.694337 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:46.694045    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:46.694337 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:46.694162    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:46.695134 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:46.694538    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:46.695134 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:46.694641    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:47.923634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:47.923602    2549 generic.go:356] "Generic (PLEG): container finished" podID="b63fbb6e-0ff3-4e2c-96f8-2717feec7816" containerID="302e404c1ef7111de93629c0e57cace0b5c3715ae0b52f9cb5b56d288a0c3075" exitCode=0
Jul 27 21:45:47.924042 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:47.923650    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9s8g" event={"ID":"b63fbb6e-0ff3-4e2c-96f8-2717feec7816","Type":"ContainerDied","Data":"302e404c1ef7111de93629c0e57cace0b5c3715ae0b52f9cb5b56d288a0c3075"}
Jul 27 21:45:48.694844 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:48.694806    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:48.694844 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:48.694827    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:48.695108 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:48.694834    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:48.695108 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:48.694965    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:48.695108 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:48.695086    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:48.695310 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:48.695158    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:49.326838 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:49.326798    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:49.327364 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:49.327037    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:49.327364 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:49.327122    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret podName:4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:21.327100185 +0000 UTC m=+66.138447537 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret") pod "global-pull-secret-syncer-8tk6w" (UID: "4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:49.427546 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:49.427504    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs\") pod \"network-metrics-daemon-hpzp8\" (UID: \"a03739ad-aa56-494b-9c4c-bd038c23af2a\") " pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:49.427772 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:49.427657    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:49.427772 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:49.427732    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs podName:a03739ad-aa56-494b-9c4c-bd038c23af2a nodeName:}" failed. No retries permitted until 2026-07-27 21:46:21.427709622 +0000 UTC m=+66.239056975 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs") pod "network-metrics-daemon-hpzp8" (UID: "a03739ad-aa56-494b-9c4c-bd038c23af2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:49.528698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:49.528662    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8629\" (UniqueName: \"kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629\") pod \"network-check-target-fqsbg\" (UID: \"b2eea199-27b2-47fc-8a3e-5145f9b9b54a\") " pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:49.528865 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:49.528840    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:45:49.528865 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:49.528855    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:45:49.528865 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:49.528865    2549 projected.go:196] Error preparing data for projected volume kube-api-access-d8629 for pod openshift-network-diagnostics/network-check-target-fqsbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:49.529021 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:49.528930    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629 podName:b2eea199-27b2-47fc-8a3e-5145f9b9b54a nodeName:}" failed. No retries permitted until 2026-07-27 21:46:21.528911569 +0000 UTC m=+66.340258960 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8629" (UniqueName: "kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629") pod "network-check-target-fqsbg" (UID: "b2eea199-27b2-47fc-8a3e-5145f9b9b54a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:50.694237 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:50.694192    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:50.694716 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:50.694254    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:50.694716 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:50.694261    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:50.694716 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:50.694336    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tk6w" podUID="4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2"
Jul 27 21:45:50.694716 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:50.694435    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpzp8" podUID="a03739ad-aa56-494b-9c4c-bd038c23af2a"
Jul 27 21:45:50.694716 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:50.694517    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqsbg" podUID="b2eea199-27b2-47fc-8a3e-5145f9b9b54a"
Jul 27 21:45:51.973275 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:51.973178    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-124.ec2.internal" event="NodeReady"
Jul 27 21:45:51.973671 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:51.973351    2549 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 21:45:52.012148 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.011355    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl"]
Jul 27 21:45:52.015261 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.015214    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl"
Jul 27 21:45:52.017316 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.017284    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74db6cf7db-lk5q4"]
Jul 27 21:45:52.018452 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.018430    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 27 21:45:52.019056 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.019039    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 27 21:45:52.019586 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.019566    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 27 21:45:52.019788 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.019767    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-824mn\""
Jul 27 21:45:52.019986 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.019967    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:45:52.021685 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.020886    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc"]
Jul 27 21:45:52.021685 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.021019    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.023538 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.022665    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"]
Jul 27 21:45:52.023538 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.022737    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc"
Jul 27 21:45:52.023873 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.023815    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 21:45:52.023980 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.023820    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8f42l\""
Jul 27 21:45:52.024506 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.024486    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-4sp48"]
Jul 27 21:45:52.024589 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.024574    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"
Jul 27 21:45:52.025186 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.025166    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 21:45:52.027004 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.025398    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 21:45:52.027004 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.026115    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9glvr"]
Jul 27 21:45:52.027004 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.026251    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:45:52.027199 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.027109    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:45:52.027638 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.027337    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 27 21:45:52.027638 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.027455    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ccqdr\""
Jul 27 21:45:52.027638 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.027474    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 27 21:45:52.027638 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.027611    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 27 21:45:52.029048 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.029029    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:45:52.029153 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.029136    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9glvr"
Jul 27 21:45:52.029331 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.029310    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 27 21:45:52.030988 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.030966    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cmbrq\""
Jul 27 21:45:52.031481 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.031376    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 27 21:45:52.038761 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.038323    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 27 21:45:52.040304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.039147    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cc4hf\""
Jul 27 21:45:52.042048 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.041612    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:45:52.042048 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.041817    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 27 21:45:52.042536 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.042371    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 27 21:45:52.043538 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.042784    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 21:45:52.043538 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.043480    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 27 21:45:52.043721 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.043700    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-j7s4m\""
Jul 27 21:45:52.043893 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.043878    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:45:52.045922 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.045784    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl"]
Jul 27 21:45:52.046912 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.046880    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc"]
Jul 27 21:45:52.048751 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.048462    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-ltzm4"]
Jul 27 21:45:52.049468 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.049446    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 27 21:45:52.052114 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.052095    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9glvr"]
Jul 27 21:45:52.052247 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.052128    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"]
Jul 27 21:45:52.052247 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.052154    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74db6cf7db-lk5q4"]
Jul 27 21:45:52.052247 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.052173    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-4sp48"]
Jul 27 21:45:52.052399 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.052288    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ltzm4"
Jul 27 21:45:52.054297 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.054279    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bfc7z"]
Jul 27 21:45:52.057756 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.055693    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9jbgg\""
Jul 27 21:45:52.057756 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.055710    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 21:45:52.057756 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.056126    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 21:45:52.059904 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.058875    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dpv6j"]
Jul 27 21:45:52.059904 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.059870    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bfc7z"
Jul 27 21:45:52.061946 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.061913    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-ltzm4"]
Jul 27 21:45:52.062045 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.061973    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bfc7z"]
Jul 27 21:45:52.062045 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.061997    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dpv6j"]
Jul 27 21:45:52.062252 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.062237    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:52.063742 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.063722    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 21:45:52.064042 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.063767    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 21:45:52.064122 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.063824    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b6tfk\""
Jul 27 21:45:52.064618 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.064582    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 21:45:52.064790 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.064760    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 21:45:52.065017 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.065004    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jb57h\""
Jul 27 21:45:52.065168 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.065148    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 21:45:52.149924 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.149758    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qm26\" (UniqueName: \"kubernetes.io/projected/69b53e4c-dae3-4de9-b6a8-4de71c24edd6-kube-api-access-2qm26\") pod \"volume-data-source-validator-549c6d54cf-9glvr\" (UID: \"69b53e4c-dae3-4de9-b6a8-4de71c24edd6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9glvr"
Jul 27 21:45:52.150032 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.149944    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc-config\") pod \"service-ca-operator-5d6c75f84f-ptvcl\" (UID: \"67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl"
Jul 27 21:45:52.150032 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.149969    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-config-volume\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:52.150032 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.149996    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9593fc04-69be-49b6-b0d1-dda41059ce61-ca-trust-extracted\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.150032 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150019    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tzl\" (UniqueName: \"kubernetes.io/projected/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-kube-api-access-h5tzl\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:52.150203 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150072    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9593fc04-69be-49b6-b0d1-dda41059ce61-installation-pull-secrets\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.150203 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150097    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bjz6\" (UniqueName: \"kubernetes.io/projected/f2576233-1733-4b8a-9056-09acb732b3d0-kube-api-access-8bjz6\") pod \"ingress-canary-bfc7z\" (UID: \"f2576233-1733-4b8a-9056-09acb732b3d0\") " pod="openshift-ingress-canary/ingress-canary-bfc7z"
Jul 27 21:45:52.150203 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150118    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4bnsr\" (UID: \"b7ec81ff-7926-4a88-89c3-ab110245f6e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"
Jul 27 21:45:52.150203 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150144    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9593fc04-69be-49b6-b0d1-dda41059ce61-image-registry-private-configuration\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.150203 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150172    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ptvcl\" (UID: \"67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl"
Jul 27 21:45:52.150385 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150271    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:52.150385 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150288    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c7b3d47-576c-4657-a558-69c78baeebad-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc\" (UID: \"5c7b3d47-576c-4657-a558-69c78baeebad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc"
Jul 27 21:45:52.150385 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150307    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1662a7c-ddf5-4b76-b4cf-57c34f091232-serving-cert\") pod \"console-operator-584dcb6b54-4sp48\" (UID: \"b1662a7c-ddf5-4b76-b4cf-57c34f091232\") " pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:45:52.150385 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150323    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tksnv\" (UniqueName: \"kubernetes.io/projected/5c7b3d47-576c-4657-a558-69c78baeebad-kube-api-access-tksnv\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc\" (UID: \"5c7b3d47-576c-4657-a558-69c78baeebad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc"
Jul 27 21:45:52.150385 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150359    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-828x8\" (UniqueName: \"kubernetes.io/projected/67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc-kube-api-access-828x8\") pod \"service-ca-operator-5d6c75f84f-ptvcl\" (UID: \"67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl"
Jul 27 21:45:52.150561 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150388    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9593fc04-69be-49b6-b0d1-dda41059ce61-trusted-ca\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.150561 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150405    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1662a7c-ddf5-4b76-b4cf-57c34f091232-trusted-ca\") pod \"console-operator-584dcb6b54-4sp48\" (UID: \"b1662a7c-ddf5-4b76-b4cf-57c34f091232\") " pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:45:52.150561 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150448    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1662a7c-ddf5-4b76-b4cf-57c34f091232-config\") pod \"console-operator-584dcb6b54-4sp48\" (UID: \"b1662a7c-ddf5-4b76-b4cf-57c34f091232\") " pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:45:52.150561 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150502    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wgzz\" (UniqueName: \"kubernetes.io/projected/b1662a7c-ddf5-4b76-b4cf-57c34f091232-kube-api-access-8wgzz\") pod \"console-operator-584dcb6b54-4sp48\" (UID: \"b1662a7c-ddf5-4b76-b4cf-57c34f091232\") " pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:45:52.150561 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150551    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-tmp-dir\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:52.150701 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150574    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c7b3d47-576c-4657-a558-69c78baeebad-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc\" (UID: \"5c7b3d47-576c-4657-a558-69c78baeebad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc"
Jul 27 21:45:52.150701 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150608    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2576233-1733-4b8a-9056-09acb732b3d0-cert\") pod \"ingress-canary-bfc7z\" (UID: \"f2576233-1733-4b8a-9056-09acb732b3d0\") " pod="openshift-ingress-canary/ingress-canary-bfc7z"
Jul 27 21:45:52.150701 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150630    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxw66\" (UniqueName: \"kubernetes.io/projected/b7ec81ff-7926-4a88-89c3-ab110245f6e7-kube-api-access-jxw66\") pod \"cluster-samples-operator-6748489656-4bnsr\" (UID: \"b7ec81ff-7926-4a88-89c3-ab110245f6e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"
Jul 27 21:45:52.150701 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150682    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-certificates\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.150853 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150714    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.150853 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150728    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-bound-sa-token\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.150853 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150745    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv77p\" (UniqueName: \"kubernetes.io/projected/e71e0949-4140-4254-89a0-4cf9eea01372-kube-api-access-dv77p\") pod \"network-check-source-7d88cb55d8-ltzm4\" (UID: \"e71e0949-4140-4254-89a0-4cf9eea01372\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ltzm4"
Jul 27 21:45:52.150853 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.150767    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74h2c\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-kube-api-access-74h2c\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.251738 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.251709    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9593fc04-69be-49b6-b0d1-dda41059ce61-installation-pull-secrets\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.251894 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.251744    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bjz6\" (UniqueName: \"kubernetes.io/projected/f2576233-1733-4b8a-9056-09acb732b3d0-kube-api-access-8bjz6\") pod \"ingress-canary-bfc7z\" (UID: \"f2576233-1733-4b8a-9056-09acb732b3d0\") " pod="openshift-ingress-canary/ingress-canary-bfc7z"
Jul 27 21:45:52.251894 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.251768    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4bnsr\" (UID: \"b7ec81ff-7926-4a88-89c3-ab110245f6e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"
Jul 27 21:45:52.251894 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.251809    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9593fc04-69be-49b6-b0d1-dda41059ce61-image-registry-private-configuration\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.251894 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.251838    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ptvcl\" (UID: \"67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl"
Jul 27 21:45:52.251894 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.251865    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:52.251894 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.251889    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c7b3d47-576c-4657-a558-69c78baeebad-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc\" (UID: \"5c7b3d47-576c-4657-a558-69c78baeebad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc"
Jul 27 21:45:52.252269 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.251917    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1662a7c-ddf5-4b76-b4cf-57c34f091232-serving-cert\") pod \"console-operator-584dcb6b54-4sp48\" (UID: \"b1662a7c-ddf5-4b76-b4cf-57c34f091232\") " pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:45:52.252269 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.251943    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tksnv\" (UniqueName: \"kubernetes.io/projected/5c7b3d47-576c-4657-a558-69c78baeebad-kube-api-access-tksnv\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc\" (UID: \"5c7b3d47-576c-4657-a558-69c78baeebad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc"
Jul 27 21:45:52.252269 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.251973    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828x8\" (UniqueName: \"kubernetes.io/projected/67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc-kube-api-access-828x8\") pod \"service-ca-operator-5d6c75f84f-ptvcl\" (UID: \"67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl"
Jul 27 21:45:52.252269 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:52.251982    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 21:45:52.252269 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.252003    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9593fc04-69be-49b6-b0d1-dda41059ce61-trusted-ca\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.252269 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.252028    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1662a7c-ddf5-4b76-b4cf-57c34f091232-trusted-ca\") pod \"console-operator-584dcb6b54-4sp48\" (UID: \"b1662a7c-ddf5-4b76-b4cf-57c34f091232\") " pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:45:52.252269 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.252056    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1662a7c-ddf5-4b76-b4cf-57c34f091232-config\") pod \"console-operator-584dcb6b54-4sp48\" (UID: \"b1662a7c-ddf5-4b76-b4cf-57c34f091232\") " pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:45:52.252269 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:52.252081    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls podName:b7ec81ff-7926-4a88-89c3-ab110245f6e7 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:52.75205789 +0000 UTC m=+37.563405255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls") pod "cluster-samples-operator-6748489656-4bnsr" (UID: "b7ec81ff-7926-4a88-89c3-ab110245f6e7") : secret "samples-operator-tls" not found
Jul 27 21:45:52.252269 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.252132    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wgzz\" (UniqueName: \"kubernetes.io/projected/b1662a7c-ddf5-4b76-b4cf-57c34f091232-kube-api-access-8wgzz\") pod \"console-operator-584dcb6b54-4sp48\" (UID: \"b1662a7c-ddf5-4b76-b4cf-57c34f091232\") " pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:45:52.252269 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.252183    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-tmp-dir\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:52.252269 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.252211    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c7b3d47-576c-4657-a558-69c78baeebad-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc\" (UID: \"5c7b3d47-576c-4657-a558-69c78baeebad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc"
Jul 27 21:45:52.252269 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.252252    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2576233-1733-4b8a-9056-09acb732b3d0-cert\") pod \"ingress-canary-bfc7z\" (UID: \"f2576233-1733-4b8a-9056-09acb732b3d0\") " pod="openshift-ingress-canary/ingress-canary-bfc7z"
Jul 27 21:45:52.252269 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.252277    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxw66\" (UniqueName: \"kubernetes.io/projected/b7ec81ff-7926-4a88-89c3-ab110245f6e7-kube-api-access-jxw66\") pod \"cluster-samples-operator-6748489656-4bnsr\" (UID: \"b7ec81ff-7926-4a88-89c3-ab110245f6e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"
Jul 27 21:45:52.252845 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.252638    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c7b3d47-576c-4657-a558-69c78baeebad-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc\" (UID: \"5c7b3d47-576c-4657-a558-69c78baeebad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc"
Jul 27 21:45:52.252845 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.252764    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1662a7c-ddf5-4b76-b4cf-57c34f091232-config\") pod \"console-operator-584dcb6b54-4sp48\" (UID: \"b1662a7c-ddf5-4b76-b4cf-57c34f091232\") " pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:45:52.252949 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:52.252851    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:45:52.252949 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:52.252894    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls podName:2c5933c6-64fd-4d0d-b8f9-c32fd675d89d nodeName:}" failed. No retries permitted until 2026-07-27 21:45:52.752879162 +0000 UTC m=+37.564226512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls") pod "dns-default-dpv6j" (UID: "2c5933c6-64fd-4d0d-b8f9-c32fd675d89d") : secret "dns-default-metrics-tls" not found
Jul 27 21:45:52.252949 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.252928    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-tmp-dir\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:52.253372 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.253346    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1662a7c-ddf5-4b76-b4cf-57c34f091232-trusted-ca\") pod \"console-operator-584dcb6b54-4sp48\" (UID: \"b1662a7c-ddf5-4b76-b4cf-57c34f091232\") " pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:45:52.253502 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.253401    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9593fc04-69be-49b6-b0d1-dda41059ce61-trusted-ca\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.253981 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.253536    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-certificates\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.253981 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.253745    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.253981 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.253777    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-bound-sa-token\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.253981 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.253804    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv77p\" (UniqueName: \"kubernetes.io/projected/e71e0949-4140-4254-89a0-4cf9eea01372-kube-api-access-dv77p\") pod \"network-check-source-7d88cb55d8-ltzm4\" (UID: \"e71e0949-4140-4254-89a0-4cf9eea01372\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ltzm4"
Jul 27 21:45:52.253981 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.253843    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74h2c\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-kube-api-access-74h2c\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.253981 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.253894    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qm26\" (UniqueName: \"kubernetes.io/projected/69b53e4c-dae3-4de9-b6a8-4de71c24edd6-kube-api-access-2qm26\") pod \"volume-data-source-validator-549c6d54cf-9glvr\" (UID: \"69b53e4c-dae3-4de9-b6a8-4de71c24edd6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9glvr"
Jul 27 21:45:52.253981 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.253927    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc-config\") pod \"service-ca-operator-5d6c75f84f-ptvcl\" (UID: \"67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl"
Jul 27 21:45:52.253981 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.253953    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-config-volume\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:52.253981 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.253959    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-certificates\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.253981 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.253982    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9593fc04-69be-49b6-b0d1-dda41059ce61-ca-trust-extracted\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.254513 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.254009    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tzl\" (UniqueName: \"kubernetes.io/projected/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-kube-api-access-h5tzl\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:52.254513 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:52.254280    2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:45:52.254513 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:52.254296    2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74db6cf7db-lk5q4: secret "image-registry-tls" not found
Jul 27 21:45:52.254513 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:52.254342    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls podName:9593fc04-69be-49b6-b0d1-dda41059ce61 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:52.754325936 +0000 UTC m=+37.565673292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls") pod "image-registry-74db6cf7db-lk5q4" (UID: "9593fc04-69be-49b6-b0d1-dda41059ce61") : secret "image-registry-tls" not found
Jul 27 21:45:52.254970 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.254942    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9593fc04-69be-49b6-b0d1-dda41059ce61-ca-trust-extracted\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.255839 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.255368    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc-config\") pod \"service-ca-operator-5d6c75f84f-ptvcl\" (UID: \"67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl"
Jul 27 21:45:52.255839 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.255796    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-config-volume\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:52.257697 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.257643    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c7b3d47-576c-4657-a558-69c78baeebad-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc\" (UID: \"5c7b3d47-576c-4657-a558-69c78baeebad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc"
Jul 27 21:45:52.257697 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.257672    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1662a7c-ddf5-4b76-b4cf-57c34f091232-serving-cert\") pod \"console-operator-584dcb6b54-4sp48\" (UID: \"b1662a7c-ddf5-4b76-b4cf-57c34f091232\") " pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:45:52.257805 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.257789    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ptvcl\" (UID: \"67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl"
Jul 27 21:45:52.257841 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.257814    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9593fc04-69be-49b6-b0d1-dda41059ce61-image-registry-private-configuration\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.257966 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.257950    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2576233-1733-4b8a-9056-09acb732b3d0-cert\") pod \"ingress-canary-bfc7z\" (UID: \"f2576233-1733-4b8a-9056-09acb732b3d0\") " pod="openshift-ingress-canary/ingress-canary-bfc7z"
Jul 27 21:45:52.258251 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.258208    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9593fc04-69be-49b6-b0d1-dda41059ce61-installation-pull-secrets\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.261996 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.261970    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bjz6\" (UniqueName: \"kubernetes.io/projected/f2576233-1733-4b8a-9056-09acb732b3d0-kube-api-access-8bjz6\") pod \"ingress-canary-bfc7z\" (UID: \"f2576233-1733-4b8a-9056-09acb732b3d0\") " pod="openshift-ingress-canary/ingress-canary-bfc7z"
Jul 27 21:45:52.262317 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.262296    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tksnv\" (UniqueName: \"kubernetes.io/projected/5c7b3d47-576c-4657-a558-69c78baeebad-kube-api-access-tksnv\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc\" (UID: \"5c7b3d47-576c-4657-a558-69c78baeebad\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc"
Jul 27 21:45:52.263572 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.263549    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74h2c\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-kube-api-access-74h2c\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.264281 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.263763    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxw66\" (UniqueName: \"kubernetes.io/projected/b7ec81ff-7926-4a88-89c3-ab110245f6e7-kube-api-access-jxw66\") pod \"cluster-samples-operator-6748489656-4bnsr\" (UID: \"b7ec81ff-7926-4a88-89c3-ab110245f6e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"
Jul 27 21:45:52.264281 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.264081    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-828x8\" (UniqueName: \"kubernetes.io/projected/67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc-kube-api-access-828x8\") pod \"service-ca-operator-5d6c75f84f-ptvcl\" (UID: \"67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl"
Jul 27 21:45:52.264568 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.264544    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wgzz\" (UniqueName: \"kubernetes.io/projected/b1662a7c-ddf5-4b76-b4cf-57c34f091232-kube-api-access-8wgzz\") pod \"console-operator-584dcb6b54-4sp48\" (UID: \"b1662a7c-ddf5-4b76-b4cf-57c34f091232\") " pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:45:52.264667 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.264543    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-bound-sa-token\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.264918 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.264892    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qm26\" (UniqueName: \"kubernetes.io/projected/69b53e4c-dae3-4de9-b6a8-4de71c24edd6-kube-api-access-2qm26\") pod \"volume-data-source-validator-549c6d54cf-9glvr\" (UID: \"69b53e4c-dae3-4de9-b6a8-4de71c24edd6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9glvr"
Jul 27 21:45:52.265589 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.265568    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv77p\" (UniqueName: \"kubernetes.io/projected/e71e0949-4140-4254-89a0-4cf9eea01372-kube-api-access-dv77p\") pod \"network-check-source-7d88cb55d8-ltzm4\" (UID: \"e71e0949-4140-4254-89a0-4cf9eea01372\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ltzm4"
Jul 27 21:45:52.265660 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.265577    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tzl\" (UniqueName: \"kubernetes.io/projected/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-kube-api-access-h5tzl\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:52.343972 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.343946    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl"
Jul 27 21:45:52.370518 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.370488    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc"
Jul 27 21:45:52.385733 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.385706    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:45:52.392833 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.392806    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9glvr"
Jul 27 21:45:52.401179 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.400994    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ltzm4"
Jul 27 21:45:52.409579 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.408441    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bfc7z"
Jul 27 21:45:52.521340 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.520995    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl"]
Jul 27 21:45:52.572022 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.570916    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc"]
Jul 27 21:45:52.576788 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:52.576733    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7b3d47_576c_4657_a558_69c78baeebad.slice/crio-b172a0a862b2d8c89df5f1f4e4ff31bb1564d64460910adfc5f1700bce2daeb4 WatchSource:0}: Error finding container b172a0a862b2d8c89df5f1f4e4ff31bb1564d64460910adfc5f1700bce2daeb4: Status 404 returned error can't find the container with id b172a0a862b2d8c89df5f1f4e4ff31bb1564d64460910adfc5f1700bce2daeb4
Jul 27 21:45:52.609253 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.609199    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9glvr"]
Jul 27 21:45:52.612418 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:52.612379    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b53e4c_dae3_4de9_b6a8_4de71c24edd6.slice/crio-984a6fecfba1c9aa474b15a4e09d9e075004e0a95a63685bdfb96653ef0fe029 WatchSource:0}: Error finding container 984a6fecfba1c9aa474b15a4e09d9e075004e0a95a63685bdfb96653ef0fe029: Status 404 returned error can't find the container with id 984a6fecfba1c9aa474b15a4e09d9e075004e0a95a63685bdfb96653ef0fe029
Jul 27 21:45:52.615245 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.615200    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-4sp48"]
Jul 27 21:45:52.618461 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:52.618425    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1662a7c_ddf5_4b76_b4cf_57c34f091232.slice/crio-aa252cab8c1a272c1303cbb1a25d30a1ac97f9b0095e3af1aa4cbfc198b65aba WatchSource:0}: Error finding container aa252cab8c1a272c1303cbb1a25d30a1ac97f9b0095e3af1aa4cbfc198b65aba: Status 404 returned error can't find the container with id aa252cab8c1a272c1303cbb1a25d30a1ac97f9b0095e3af1aa4cbfc198b65aba
Jul 27 21:45:52.632567 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.632533    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bfc7z"]
Jul 27 21:45:52.635249 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:52.635210    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2576233_1733_4b8a_9056_09acb732b3d0.slice/crio-fa048256d4768df8ac8c40a32fcc1d06a6b0ad71bfe9ec1349df4aefb8724897 WatchSource:0}: Error finding container fa048256d4768df8ac8c40a32fcc1d06a6b0ad71bfe9ec1349df4aefb8724897: Status 404 returned error can't find the container with id fa048256d4768df8ac8c40a32fcc1d06a6b0ad71bfe9ec1349df4aefb8724897
Jul 27 21:45:52.644139 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.644117    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-ltzm4"]
Jul 27 21:45:52.647497 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:45:52.647468    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71e0949_4140_4254_89a0_4cf9eea01372.slice/crio-db50a3fcca2bd25323499a93aea729433e8ff4938c9a7e5a8d943637a051c0ed WatchSource:0}: Error finding container db50a3fcca2bd25323499a93aea729433e8ff4938c9a7e5a8d943637a051c0ed: Status 404 returned error can't find the container with id db50a3fcca2bd25323499a93aea729433e8ff4938c9a7e5a8d943637a051c0ed
Jul 27 21:45:52.694760 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.694719    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:45:52.694943 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.694731    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:45:52.694943 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.694895    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:45:52.697695 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.697678    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 21:45:52.697695 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.697692    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kt796\""
Jul 27 21:45:52.697852 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.697700    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cj7gv\""
Jul 27 21:45:52.697852 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.697698    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 21:45:52.760283 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.760178    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4bnsr\" (UID: \"b7ec81ff-7926-4a88-89c3-ab110245f6e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"
Jul 27 21:45:52.760283 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.760218    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:52.760509 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.760301    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:52.760509 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:52.760327    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 21:45:52.760509 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:52.760397    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls podName:b7ec81ff-7926-4a88-89c3-ab110245f6e7 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:53.760378161 +0000 UTC m=+38.571725516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls") pod "cluster-samples-operator-6748489656-4bnsr" (UID: "b7ec81ff-7926-4a88-89c3-ab110245f6e7") : secret "samples-operator-tls" not found
Jul 27 21:45:52.760509 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:52.760415    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:45:52.760509 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:52.760437    2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:45:52.760509 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:52.760448    2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74db6cf7db-lk5q4: secret "image-registry-tls" not found
Jul 27 21:45:52.760509 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:52.760471    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls podName:2c5933c6-64fd-4d0d-b8f9-c32fd675d89d nodeName:}" failed. No retries permitted until 2026-07-27 21:45:53.760454823 +0000 UTC m=+38.571802174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls") pod "dns-default-dpv6j" (UID: "2c5933c6-64fd-4d0d-b8f9-c32fd675d89d") : secret "dns-default-metrics-tls" not found
Jul 27 21:45:52.760509 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:52.760491    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls podName:9593fc04-69be-49b6-b0d1-dda41059ce61 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:53.76048196 +0000 UTC m=+38.571829312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls") pod "image-registry-74db6cf7db-lk5q4" (UID: "9593fc04-69be-49b6-b0d1-dda41059ce61") : secret "image-registry-tls" not found
Jul 27 21:45:52.936664 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.936628    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bfc7z" event={"ID":"f2576233-1733-4b8a-9056-09acb732b3d0","Type":"ContainerStarted","Data":"fa048256d4768df8ac8c40a32fcc1d06a6b0ad71bfe9ec1349df4aefb8724897"}
Jul 27 21:45:52.937660 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.937636    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9glvr" event={"ID":"69b53e4c-dae3-4de9-b6a8-4de71c24edd6","Type":"ContainerStarted","Data":"984a6fecfba1c9aa474b15a4e09d9e075004e0a95a63685bdfb96653ef0fe029"}
Jul 27 21:45:52.938698 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.938667    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc" event={"ID":"5c7b3d47-576c-4657-a558-69c78baeebad","Type":"ContainerStarted","Data":"b172a0a862b2d8c89df5f1f4e4ff31bb1564d64460910adfc5f1700bce2daeb4"}
Jul 27 21:45:52.939761 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.939740    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl" event={"ID":"67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc","Type":"ContainerStarted","Data":"a3d453dee304ae588cc873cd971d21864d0ee729990c2680ab7841de96fdfe58"}
Jul 27 21:45:52.942383 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.942361    2549 generic.go:356] "Generic (PLEG): container finished" podID="b63fbb6e-0ff3-4e2c-96f8-2717feec7816" containerID="97828c03c8670667d6481b06cddd4c62798229e8497b1fbbb4d551a6cd963faf" exitCode=0
Jul 27 21:45:52.942460 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.942419    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9s8g" event={"ID":"b63fbb6e-0ff3-4e2c-96f8-2717feec7816","Type":"ContainerDied","Data":"97828c03c8670667d6481b06cddd4c62798229e8497b1fbbb4d551a6cd963faf"}
Jul 27 21:45:52.943578 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.943560    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ltzm4" event={"ID":"e71e0949-4140-4254-89a0-4cf9eea01372","Type":"ContainerStarted","Data":"db50a3fcca2bd25323499a93aea729433e8ff4938c9a7e5a8d943637a051c0ed"}
Jul 27 21:45:52.944632 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:52.944612    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-4sp48" event={"ID":"b1662a7c-ddf5-4b76-b4cf-57c34f091232","Type":"ContainerStarted","Data":"aa252cab8c1a272c1303cbb1a25d30a1ac97f9b0095e3af1aa4cbfc198b65aba"}
Jul 27 21:45:53.693818 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.692969    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"]
Jul 27 21:45:53.699309 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.696697    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"
Jul 27 21:45:53.703630 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.703578    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-lhp6l\""
Jul 27 21:45:53.703767 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.703604    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 27 21:45:53.707375 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.705204    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 27 21:45:53.707375 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.706779    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"]
Jul 27 21:45:53.770663 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.770628    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:53.770853 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.770705    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4bnsr\" (UID: \"b7ec81ff-7926-4a88-89c3-ab110245f6e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"
Jul 27 21:45:53.770853 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.770737    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:53.770974 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:53.770865    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:45:53.770974 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:53.770939    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls podName:2c5933c6-64fd-4d0d-b8f9-c32fd675d89d nodeName:}" failed. No retries permitted until 2026-07-27 21:45:55.77091941 +0000 UTC m=+40.582266780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls") pod "dns-default-dpv6j" (UID: "2c5933c6-64fd-4d0d-b8f9-c32fd675d89d") : secret "dns-default-metrics-tls" not found
Jul 27 21:45:53.771440 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:53.771418    2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:45:53.771440 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:53.771439    2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74db6cf7db-lk5q4: secret "image-registry-tls" not found
Jul 27 21:45:53.771623 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:53.771486    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls podName:9593fc04-69be-49b6-b0d1-dda41059ce61 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:55.771471102 +0000 UTC m=+40.582818451 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls") pod "image-registry-74db6cf7db-lk5q4" (UID: "9593fc04-69be-49b6-b0d1-dda41059ce61") : secret "image-registry-tls" not found
Jul 27 21:45:53.771623 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:53.771550    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 21:45:53.771623 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:53.771586    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls podName:b7ec81ff-7926-4a88-89c3-ab110245f6e7 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:55.771575333 +0000 UTC m=+40.582922683 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls") pod "cluster-samples-operator-6748489656-4bnsr" (UID: "b7ec81ff-7926-4a88-89c3-ab110245f6e7") : secret "samples-operator-tls" not found
Jul 27 21:45:53.871634 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.871591    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bf6dz\" (UID: \"ebc50b72-b32f-42b2-b817-09d4ed57857b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"
Jul 27 21:45:53.871828 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.871774    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ebc50b72-b32f-42b2-b817-09d4ed57857b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bf6dz\" (UID: \"ebc50b72-b32f-42b2-b817-09d4ed57857b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"
Jul 27 21:45:53.953816 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.952815    2549 generic.go:356] "Generic (PLEG): container finished" podID="b63fbb6e-0ff3-4e2c-96f8-2717feec7816" containerID="34042d04580b37be075128d759784fa5ec04ede897974540c63458ca412e6e89" exitCode=0
Jul 27 21:45:53.953816 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.952882    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9s8g" event={"ID":"b63fbb6e-0ff3-4e2c-96f8-2717feec7816","Type":"ContainerDied","Data":"34042d04580b37be075128d759784fa5ec04ede897974540c63458ca412e6e89"}
Jul 27 21:45:53.972661 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.972614    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ebc50b72-b32f-42b2-b817-09d4ed57857b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bf6dz\" (UID: \"ebc50b72-b32f-42b2-b817-09d4ed57857b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"
Jul 27 21:45:53.972821 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.972694    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bf6dz\" (UID: \"ebc50b72-b32f-42b2-b817-09d4ed57857b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"
Jul 27 21:45:53.972908 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:53.972892    2549 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 21:45:53.973823 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:53.972962    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert podName:ebc50b72-b32f-42b2-b817-09d4ed57857b nodeName:}" failed. No retries permitted until 2026-07-27 21:45:54.47293877 +0000 UTC m=+39.284286346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bf6dz" (UID: "ebc50b72-b32f-42b2-b817-09d4ed57857b") : secret "networking-console-plugin-cert" not found
Jul 27 21:45:53.973823 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:53.973772    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ebc50b72-b32f-42b2-b817-09d4ed57857b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bf6dz\" (UID: \"ebc50b72-b32f-42b2-b817-09d4ed57857b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"
Jul 27 21:45:54.481337 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:54.481286    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bf6dz\" (UID: \"ebc50b72-b32f-42b2-b817-09d4ed57857b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"
Jul 27 21:45:54.481553 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:54.481535    2549 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 21:45:54.481669 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:54.481656    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert podName:ebc50b72-b32f-42b2-b817-09d4ed57857b nodeName:}" failed. No retries permitted until 2026-07-27 21:45:55.481591559 +0000 UTC m=+40.292938915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bf6dz" (UID: "ebc50b72-b32f-42b2-b817-09d4ed57857b") : secret "networking-console-plugin-cert" not found
Jul 27 21:45:54.964783 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:54.964435    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9s8g" event={"ID":"b63fbb6e-0ff3-4e2c-96f8-2717feec7816","Type":"ContainerStarted","Data":"d4635c2ad3b39f84c83a6de6666688ebc848243771e266f428fc3e5620102827"}
Jul 27 21:45:54.968093 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:54.968018    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9glvr" event={"ID":"69b53e4c-dae3-4de9-b6a8-4de71c24edd6","Type":"ContainerStarted","Data":"bee0a9543cb2a479a3af688c28c01db7f053ae5f2b44595cd24c2345257199d7"}
Jul 27 21:45:54.989845 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:54.989779    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b9s8g" podStartSLOduration=8.857582567 podStartE2EDuration="38.989759444s" totalImagesPullingTime="30.132176877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:54.986724429 +0000 UTC m=+39.798071805" watchObservedRunningTime="2026-07-27 21:45:54.989759444 +0000 UTC m=+39.801106817"
Jul 27 21:45:55.017726 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:55.017460    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9glvr" podStartSLOduration=36.144382276 podStartE2EDuration="38.017437448s" totalImagesPullingTime="1.873055172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:55.016377606 +0000 UTC m=+39.827724971" watchObservedRunningTime="2026-07-27 21:45:55.017437448 +0000 UTC m=+39.828784827"
Jul 27 21:45:55.491574 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:55.491538    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bf6dz\" (UID: \"ebc50b72-b32f-42b2-b817-09d4ed57857b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"
Jul 27 21:45:55.491761 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:55.491710    2549 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 21:45:55.491833 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:55.491794    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert podName:ebc50b72-b32f-42b2-b817-09d4ed57857b nodeName:}" failed. No retries permitted until 2026-07-27 21:45:57.491772321 +0000 UTC m=+42.303119694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bf6dz" (UID: "ebc50b72-b32f-42b2-b817-09d4ed57857b") : secret "networking-console-plugin-cert" not found
Jul 27 21:45:55.793777 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:55.793687    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:55.793777 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:55.793773    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4bnsr\" (UID: \"b7ec81ff-7926-4a88-89c3-ab110245f6e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"
Jul 27 21:45:55.794032 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:55.793815    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:55.794032 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:55.793841    2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:45:55.794032 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:55.793862    2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74db6cf7db-lk5q4: secret "image-registry-tls" not found
Jul 27 21:45:55.794032 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:55.793918    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls podName:9593fc04-69be-49b6-b0d1-dda41059ce61 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:59.793902765 +0000 UTC m=+44.605250118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls") pod "image-registry-74db6cf7db-lk5q4" (UID: "9593fc04-69be-49b6-b0d1-dda41059ce61") : secret "image-registry-tls" not found
Jul 27 21:45:55.794032 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:55.793918    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:45:55.794032 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:55.793919    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 21:45:55.794032 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:55.793979    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls podName:2c5933c6-64fd-4d0d-b8f9-c32fd675d89d nodeName:}" failed. No retries permitted until 2026-07-27 21:45:59.79396465 +0000 UTC m=+44.605312003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls") pod "dns-default-dpv6j" (UID: "2c5933c6-64fd-4d0d-b8f9-c32fd675d89d") : secret "dns-default-metrics-tls" not found
Jul 27 21:45:55.794032 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:55.793996    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls podName:b7ec81ff-7926-4a88-89c3-ab110245f6e7 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:59.793987263 +0000 UTC m=+44.605334614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls") pod "cluster-samples-operator-6748489656-4bnsr" (UID: "b7ec81ff-7926-4a88-89c3-ab110245f6e7") : secret "samples-operator-tls" not found
Jul 27 21:45:57.507577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:57.507537    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bf6dz\" (UID: \"ebc50b72-b32f-42b2-b817-09d4ed57857b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"
Jul 27 21:45:57.508290 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:57.507709    2549 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 21:45:57.508290 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:57.507792    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert podName:ebc50b72-b32f-42b2-b817-09d4ed57857b nodeName:}" failed. No retries permitted until 2026-07-27 21:46:01.507773179 +0000 UTC m=+46.319120542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bf6dz" (UID: "ebc50b72-b32f-42b2-b817-09d4ed57857b") : secret "networking-console-plugin-cert" not found
Jul 27 21:45:59.828022 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:59.827980    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:45:59.828526 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:59.828082    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:45:59.828526 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:59.828127    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4bnsr\" (UID: \"b7ec81ff-7926-4a88-89c3-ab110245f6e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"
Jul 27 21:45:59.828526 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:59.828146    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:45:59.828526 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:59.828242    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls podName:2c5933c6-64fd-4d0d-b8f9-c32fd675d89d nodeName:}" failed. No retries permitted until 2026-07-27 21:46:07.82820051 +0000 UTC m=+52.639547866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls") pod "dns-default-dpv6j" (UID: "2c5933c6-64fd-4d0d-b8f9-c32fd675d89d") : secret "dns-default-metrics-tls" not found
Jul 27 21:45:59.828526 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:59.828246    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 21:45:59.828526 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:59.828252    2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:45:59.828526 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:59.828272    2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74db6cf7db-lk5q4: secret "image-registry-tls" not found
Jul 27 21:45:59.828526 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:59.828288    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls podName:b7ec81ff-7926-4a88-89c3-ab110245f6e7 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:07.828277113 +0000 UTC m=+52.639624464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls") pod "cluster-samples-operator-6748489656-4bnsr" (UID: "b7ec81ff-7926-4a88-89c3-ab110245f6e7") : secret "samples-operator-tls" not found
Jul 27 21:45:59.828526 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:45:59.828320    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls podName:9593fc04-69be-49b6-b0d1-dda41059ce61 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:07.828304454 +0000 UTC m=+52.639651820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls") pod "image-registry-74db6cf7db-lk5q4" (UID: "9593fc04-69be-49b6-b0d1-dda41059ce61") : secret "image-registry-tls" not found
Jul 27 21:45:59.984051 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:59.983953    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bfc7z" event={"ID":"f2576233-1733-4b8a-9056-09acb732b3d0","Type":"ContainerStarted","Data":"f672aba9c7f9623e589ba5bf763cde23a3295e0bd8576bc690e1d477e823121f"}
Jul 27 21:45:59.985548 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:59.985515    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc" event={"ID":"5c7b3d47-576c-4657-a558-69c78baeebad","Type":"ContainerStarted","Data":"8c70f3cc158d61b0bb7381cffd48a45cb928cc6b8ea1bdaed642a373d9d629eb"}
Jul 27 21:45:59.987326 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:59.987290    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl" event={"ID":"67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc","Type":"ContainerStarted","Data":"08eac3f55712e81cfbfa66ddbc44e750e9f37aff88807b754a1bdf5fd442e3eb"}
Jul 27 21:45:59.988881 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:59.988840    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ltzm4" event={"ID":"e71e0949-4140-4254-89a0-4cf9eea01372","Type":"ContainerStarted","Data":"9e57c1dedeb4d6a3a16d4300b27b533abb543eee597841b897b6cee736627d03"}
Jul 27 21:45:59.990505 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:59.990484    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/0.log"
Jul 27 21:45:59.990665 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:59.990525    2549 generic.go:356] "Generic (PLEG): container finished" podID="b1662a7c-ddf5-4b76-b4cf-57c34f091232" containerID="50a91ec1aed2d091f52f4f80716207577d301127fcaea57377a57ae7697ab365" exitCode=255
Jul 27 21:45:59.990665 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:59.990557    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-4sp48" event={"ID":"b1662a7c-ddf5-4b76-b4cf-57c34f091232","Type":"ContainerDied","Data":"50a91ec1aed2d091f52f4f80716207577d301127fcaea57377a57ae7697ab365"}
Jul 27 21:45:59.990780 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:45:59.990723    2549 scope.go:117] "RemoveContainer" containerID="50a91ec1aed2d091f52f4f80716207577d301127fcaea57377a57ae7697ab365"
Jul 27 21:46:00.016100 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:00.016049    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bfc7z" podStartSLOduration=1.219210586 podStartE2EDuration="8.01603039s" totalImagesPullingTime="6.796819804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:59.999881347 +0000 UTC m=+44.811228720" watchObservedRunningTime="2026-07-27 21:46:00.01603039 +0000 UTC m=+44.827377766"
Jul 27 21:46:00.036202 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:00.036139    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc" podStartSLOduration=36.181353285 podStartE2EDuration="43.036101491s" totalImagesPullingTime="6.854748206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:00.016359159 +0000 UTC m=+44.827706533" watchObservedRunningTime="2026-07-27 21:46:00.036101491 +0000 UTC m=+44.847448865"
Jul 27 21:46:00.036511 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:00.036478    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ltzm4" podStartSLOduration=26.24228259 podStartE2EDuration="33.036464541s" totalImagesPullingTime="6.794181951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:00.035105959 +0000 UTC m=+44.846453330" watchObservedRunningTime="2026-07-27 21:46:00.036464541 +0000 UTC m=+44.847811893"
Jul 27 21:46:00.052749 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:00.052696    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl" podStartSLOduration=36.152956699 podStartE2EDuration="43.052676472s" totalImagesPullingTime="6.899719773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:00.052154771 +0000 UTC m=+44.863502145" watchObservedRunningTime="2026-07-27 21:46:00.052676472 +0000 UTC m=+44.864023845"
Jul 27 21:46:00.996075 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:00.996045    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 21:46:00.996598 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:00.996581    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/0.log"
Jul 27 21:46:00.996668 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:00.996622    2549 generic.go:356] "Generic (PLEG): container finished" podID="b1662a7c-ddf5-4b76-b4cf-57c34f091232" containerID="d0305a9aa82b9b0ee408a7f282e08340f913ae3389ae7a0d99ee896bdc5d7bed" exitCode=255
Jul 27 21:46:00.996749 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:00.996726    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-4sp48" event={"ID":"b1662a7c-ddf5-4b76-b4cf-57c34f091232","Type":"ContainerDied","Data":"d0305a9aa82b9b0ee408a7f282e08340f913ae3389ae7a0d99ee896bdc5d7bed"}
Jul 27 21:46:00.996836 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:00.996823    2549 scope.go:117] "RemoveContainer" containerID="50a91ec1aed2d091f52f4f80716207577d301127fcaea57377a57ae7697ab365"
Jul 27 21:46:00.997121 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:00.997092    2549 scope.go:117] "RemoveContainer" containerID="d0305a9aa82b9b0ee408a7f282e08340f913ae3389ae7a0d99ee896bdc5d7bed"
Jul 27 21:46:00.997309 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:00.997284    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-4sp48_openshift-console-operator(b1662a7c-ddf5-4b76-b4cf-57c34f091232)\"" pod="openshift-console-operator/console-operator-584dcb6b54-4sp48" podUID="b1662a7c-ddf5-4b76-b4cf-57c34f091232"
Jul 27 21:46:01.544120 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:01.544072    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bf6dz\" (UID: \"ebc50b72-b32f-42b2-b817-09d4ed57857b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"
Jul 27 21:46:01.544323 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:01.544272    2549 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 21:46:01.544373 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:01.544342    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert podName:ebc50b72-b32f-42b2-b817-09d4ed57857b nodeName:}" failed. No retries permitted until 2026-07-27 21:46:09.544325861 +0000 UTC m=+54.355673211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bf6dz" (UID: "ebc50b72-b32f-42b2-b817-09d4ed57857b") : secret "networking-console-plugin-cert" not found
Jul 27 21:46:02.001496 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:02.001468    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 21:46:02.001897 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:02.001780    2549 scope.go:117] "RemoveContainer" containerID="d0305a9aa82b9b0ee408a7f282e08340f913ae3389ae7a0d99ee896bdc5d7bed"
Jul 27 21:46:02.001939 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:02.001921    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-4sp48_openshift-console-operator(b1662a7c-ddf5-4b76-b4cf-57c34f091232)\"" pod="openshift-console-operator/console-operator-584dcb6b54-4sp48" podUID="b1662a7c-ddf5-4b76-b4cf-57c34f091232"
Jul 27 21:46:02.385974 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:02.385932    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:46:02.386214 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:02.385998    2549 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:46:03.005170 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:03.005140    2549 scope.go:117] "RemoveContainer" containerID="d0305a9aa82b9b0ee408a7f282e08340f913ae3389ae7a0d99ee896bdc5d7bed"
Jul 27 21:46:03.005567 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:03.005326    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-4sp48_openshift-console-operator(b1662a7c-ddf5-4b76-b4cf-57c34f091232)\"" pod="openshift-console-operator/console-operator-584dcb6b54-4sp48" podUID="b1662a7c-ddf5-4b76-b4cf-57c34f091232"
Jul 27 21:46:07.894889 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:07.894852    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:46:07.895420 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:07.894924    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4bnsr\" (UID: \"b7ec81ff-7926-4a88-89c3-ab110245f6e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"
Jul 27 21:46:07.895420 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:07.895021    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 21:46:07.895420 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:07.895025    2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:46:07.895420 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:07.895046    2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74db6cf7db-lk5q4: secret "image-registry-tls" not found
Jul 27 21:46:07.895420 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:07.895071    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:46:07.895420 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:07.895088    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls podName:b7ec81ff-7926-4a88-89c3-ab110245f6e7 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:23.895070195 +0000 UTC m=+68.706417568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls") pod "cluster-samples-operator-6748489656-4bnsr" (UID: "b7ec81ff-7926-4a88-89c3-ab110245f6e7") : secret "samples-operator-tls" not found
Jul 27 21:46:07.895420 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:07.895109    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls podName:9593fc04-69be-49b6-b0d1-dda41059ce61 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:23.895099432 +0000 UTC m=+68.706446789 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls") pod "image-registry-74db6cf7db-lk5q4" (UID: "9593fc04-69be-49b6-b0d1-dda41059ce61") : secret "image-registry-tls" not found
Jul 27 21:46:07.895420 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:07.895144    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:46:07.895420 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:07.895190    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls podName:2c5933c6-64fd-4d0d-b8f9-c32fd675d89d nodeName:}" failed. No retries permitted until 2026-07-27 21:46:23.895182268 +0000 UTC m=+68.706529618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls") pod "dns-default-dpv6j" (UID: "2c5933c6-64fd-4d0d-b8f9-c32fd675d89d") : secret "dns-default-metrics-tls" not found
Jul 27 21:46:09.606143 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:09.606103    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bf6dz\" (UID: \"ebc50b72-b32f-42b2-b817-09d4ed57857b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"
Jul 27 21:46:09.606554 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:09.606254    2549 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 21:46:09.606554 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:09.606334    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert podName:ebc50b72-b32f-42b2-b817-09d4ed57857b nodeName:}" failed. No retries permitted until 2026-07-27 21:46:25.606320254 +0000 UTC m=+70.417667605 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bf6dz" (UID: "ebc50b72-b32f-42b2-b817-09d4ed57857b") : secret "networking-console-plugin-cert" not found
Jul 27 21:46:16.694494 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:16.694460    2549 scope.go:117] "RemoveContainer" containerID="d0305a9aa82b9b0ee408a7f282e08340f913ae3389ae7a0d99ee896bdc5d7bed"
Jul 27 21:46:16.936368 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:16.936332    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ggqh"
Jul 27 21:46:17.049394 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:17.049310    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 21:46:17.049536 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:17.049394    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-4sp48" event={"ID":"b1662a7c-ddf5-4b76-b4cf-57c34f091232","Type":"ContainerStarted","Data":"5654664d484098bbc74a8ba89c8b36b083d55db21af2119353ee6c84ba66921f"}
Jul 27 21:46:17.049740 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:17.049720    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:46:17.069422 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:17.069368    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-4sp48" podStartSLOduration=53.255150712 podStartE2EDuration="1m0.069349383s" totalImagesPullingTime="6.814198671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:17.068764079 +0000 UTC m=+61.880111453" watchObservedRunningTime="2026-07-27 21:46:17.069349383 +0000 UTC m=+61.880696754"
Jul 27 21:46:17.290406 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:17.290375    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-4sp48"
Jul 27 21:46:21.407073 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.407020    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:46:21.409830 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.409812    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 21:46:21.422195 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.422166    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2-original-pull-secret\") pod \"global-pull-secret-syncer-8tk6w\" (UID: \"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2\") " pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:46:21.504784 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.504746    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tk6w"
Jul 27 21:46:21.509739 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.509711    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs\") pod \"network-metrics-daemon-hpzp8\" (UID: \"a03739ad-aa56-494b-9c4c-bd038c23af2a\") " pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:46:21.512380 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.512357    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 21:46:21.523118 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.523088    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a03739ad-aa56-494b-9c4c-bd038c23af2a-metrics-certs\") pod \"network-metrics-daemon-hpzp8\" (UID: \"a03739ad-aa56-494b-9c4c-bd038c23af2a\") " pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:46:21.610704 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.610667    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8629\" (UniqueName: \"kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629\") pod \"network-check-target-fqsbg\" (UID: \"b2eea199-27b2-47fc-8a3e-5145f9b9b54a\") " pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:46:21.613599 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.613571    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8629\" (UniqueName: \"kubernetes.io/projected/b2eea199-27b2-47fc-8a3e-5145f9b9b54a-kube-api-access-d8629\") pod \"network-check-target-fqsbg\" (UID: \"b2eea199-27b2-47fc-8a3e-5145f9b9b54a\") " pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:46:21.635313 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.635266    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8tk6w"]
Jul 27 21:46:21.638814 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:46:21.638791    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6dfdf3_9d9e_473f_b1fc_0bc33d14cba2.slice/crio-68998de798fe727965398329c72e012a8912540c79cdf31d773cbf2d6a27b379 WatchSource:0}: Error finding container 68998de798fe727965398329c72e012a8912540c79cdf31d773cbf2d6a27b379: Status 404 returned error can't find the container with id 68998de798fe727965398329c72e012a8912540c79cdf31d773cbf2d6a27b379
Jul 27 21:46:21.813031 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.812943    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kt796\""
Jul 27 21:46:21.817188 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.817167    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cj7gv\""
Jul 27 21:46:21.821325 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.821309    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:46:21.825705 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.825689    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpzp8"
Jul 27 21:46:21.953729 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.953697    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fqsbg"]
Jul 27 21:46:21.957697 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:46:21.957668    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2eea199_27b2_47fc_8a3e_5145f9b9b54a.slice/crio-85f5c05cc3aa4492101098f8e16be86c8e863a868d75f8e11bb3ec06e3ee2ef3 WatchSource:0}: Error finding container 85f5c05cc3aa4492101098f8e16be86c8e863a868d75f8e11bb3ec06e3ee2ef3: Status 404 returned error can't find the container with id 85f5c05cc3aa4492101098f8e16be86c8e863a868d75f8e11bb3ec06e3ee2ef3
Jul 27 21:46:21.970585 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:21.970560    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hpzp8"]
Jul 27 21:46:21.973909 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:46:21.973886    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03739ad_aa56_494b_9c4c_bd038c23af2a.slice/crio-ddc645ed51338f0c68f2690cc30b3abe7f6c669187701851c6a9cf1d31d1057b WatchSource:0}: Error finding container ddc645ed51338f0c68f2690cc30b3abe7f6c669187701851c6a9cf1d31d1057b: Status 404 returned error can't find the container with id ddc645ed51338f0c68f2690cc30b3abe7f6c669187701851c6a9cf1d31d1057b
Jul 27 21:46:22.065053 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:22.064959    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8tk6w" event={"ID":"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2","Type":"ContainerStarted","Data":"68998de798fe727965398329c72e012a8912540c79cdf31d773cbf2d6a27b379"}
Jul 27 21:46:22.066105 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:22.066083    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hpzp8" event={"ID":"a03739ad-aa56-494b-9c4c-bd038c23af2a","Type":"ContainerStarted","Data":"ddc645ed51338f0c68f2690cc30b3abe7f6c669187701851c6a9cf1d31d1057b"}
Jul 27 21:46:22.067461 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:22.067442    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fqsbg" event={"ID":"b2eea199-27b2-47fc-8a3e-5145f9b9b54a","Type":"ContainerStarted","Data":"37d5e1ab62652697d5343c366990c87f105a053d3d663a392475053172c28331"}
Jul 27 21:46:22.067539 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:22.067474    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fqsbg" event={"ID":"b2eea199-27b2-47fc-8a3e-5145f9b9b54a","Type":"ContainerStarted","Data":"85f5c05cc3aa4492101098f8e16be86c8e863a868d75f8e11bb3ec06e3ee2ef3"}
Jul 27 21:46:22.067586 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:22.067573    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:46:22.082834 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:22.082785    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fqsbg" podStartSLOduration=66.082768287 podStartE2EDuration="1m6.082768287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:22.082362919 +0000 UTC m=+66.893710292" watchObservedRunningTime="2026-07-27 21:46:22.082768287 +0000 UTC m=+66.894115660"
Jul 27 21:46:23.640264 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.640215    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"]
Jul 27 21:46:23.672523 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.672488    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-cpdtd"]
Jul 27 21:46:23.672685 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.672652    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.675954 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.675926    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 21:46:23.676164 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.676142    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 21:46:23.676208 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.676194    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 27 21:46:23.676304 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.675929    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 27 21:46:23.676481 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.676467    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 27 21:46:23.676565 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.676148    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 27 21:46:23.676617 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.676605    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 21:46:23.688537 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.688479    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm"]
Jul 27 21:46:23.688668 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.688627    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-cpdtd"
Jul 27 21:46:23.691465 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.691443    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 21:46:23.691852 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.691833    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7px5n\""
Jul 27 21:46:23.691948 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.691861    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 21:46:23.702864 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.702764    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm"]
Jul 27 21:46:23.702864 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.702808    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-cpdtd"]
Jul 27 21:46:23.702864 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.702837    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"]
Jul 27 21:46:23.703088 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.702882    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm"
Jul 27 21:46:23.707819 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.705910    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-f7vgz\""
Jul 27 21:46:23.707819 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.705959    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 27 21:46:23.731337 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.731127    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ddf729fb-893f-4607-8835-d9505ef490f4-ca\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.731337 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.731179    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ddf729fb-893f-4607-8835-d9505ef490f4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.731337 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.731211    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf729fb-893f-4607-8835-d9505ef490f4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.731337 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.731316    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99559\" (UniqueName: \"kubernetes.io/projected/ddf729fb-893f-4607-8835-d9505ef490f4-kube-api-access-99559\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.731616 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.731402    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddf729fb-893f-4607-8835-d9505ef490f4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.731616 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.731451    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ddf729fb-893f-4607-8835-d9505ef490f4-hub\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.832791 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.832760    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ddf729fb-893f-4607-8835-d9505ef490f4-ca\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.832961 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.832808    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d57g6\" (UniqueName: \"kubernetes.io/projected/c70e3eca-af15-4b11-ab52-71b75ea220a4-kube-api-access-d57g6\") pod \"downloads-7c97dc4555-cpdtd\" (UID: \"c70e3eca-af15-4b11-ab52-71b75ea220a4\") " pod="openshift-console/downloads-7c97dc4555-cpdtd"
Jul 27 21:46:23.832961 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.832847    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ddf729fb-893f-4607-8835-d9505ef490f4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.832961 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.832872    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf729fb-893f-4607-8835-d9505ef490f4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.832961 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.832924    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/235a8e7d-4ca5-4c22-b375-ade2f0d3499f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm\" (UID: \"235a8e7d-4ca5-4c22-b375-ade2f0d3499f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm"
Jul 27 21:46:23.833202 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.832960    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99559\" (UniqueName: \"kubernetes.io/projected/ddf729fb-893f-4607-8835-d9505ef490f4-kube-api-access-99559\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.833202 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.833000    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddf729fb-893f-4607-8835-d9505ef490f4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.833202 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.833040    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ddf729fb-893f-4607-8835-d9505ef490f4-hub\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.833202 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.833074    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5tjk\" (UniqueName: \"kubernetes.io/projected/235a8e7d-4ca5-4c22-b375-ade2f0d3499f-kube-api-access-w5tjk\") pod \"managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm\" (UID: \"235a8e7d-4ca5-4c22-b375-ade2f0d3499f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm"
Jul 27 21:46:23.834150 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.833878    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf729fb-893f-4607-8835-d9505ef490f4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.836141 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.836118    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ddf729fb-893f-4607-8835-d9505ef490f4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.836320 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.836298    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ddf729fb-893f-4607-8835-d9505ef490f4-ca\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.836503 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.836474    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ddf729fb-893f-4607-8835-d9505ef490f4-hub\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.837972 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.837948    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddf729fb-893f-4607-8835-d9505ef490f4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.842798 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.842775    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99559\" (UniqueName: \"kubernetes.io/projected/ddf729fb-893f-4607-8835-d9505ef490f4-kube-api-access-99559\") pod \"cluster-proxy-proxy-agent-7f76d679b9-x2q55\" (UID: \"ddf729fb-893f-4607-8835-d9505ef490f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:23.934485 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.934450    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/235a8e7d-4ca5-4c22-b375-ade2f0d3499f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm\" (UID: \"235a8e7d-4ca5-4c22-b375-ade2f0d3499f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm"
Jul 27 21:46:23.934665 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.934501    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:46:23.934665 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.934547    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5tjk\" (UniqueName: \"kubernetes.io/projected/235a8e7d-4ca5-4c22-b375-ade2f0d3499f-kube-api-access-w5tjk\") pod \"managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm\" (UID: \"235a8e7d-4ca5-4c22-b375-ade2f0d3499f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm"
Jul 27 21:46:23.934665 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.934580    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:46:23.934665 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.934619    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d57g6\" (UniqueName: \"kubernetes.io/projected/c70e3eca-af15-4b11-ab52-71b75ea220a4-kube-api-access-d57g6\") pod \"downloads-7c97dc4555-cpdtd\" (UID: \"c70e3eca-af15-4b11-ab52-71b75ea220a4\") " pod="openshift-console/downloads-7c97dc4555-cpdtd"
Jul 27 21:46:23.934877 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.934665    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4bnsr\" (UID: \"b7ec81ff-7926-4a88-89c3-ab110245f6e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"
Jul 27 21:46:23.937558 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.937528    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c5933c6-64fd-4d0d-b8f9-c32fd675d89d-metrics-tls\") pod \"dns-default-dpv6j\" (UID: \"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d\") " pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:46:23.937724 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.937646    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls\") pod \"image-registry-74db6cf7db-lk5q4\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") " pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:46:23.937724 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.937683    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7ec81ff-7926-4a88-89c3-ab110245f6e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4bnsr\" (UID: \"b7ec81ff-7926-4a88-89c3-ab110245f6e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"
Jul 27 21:46:23.938190 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.938171    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/235a8e7d-4ca5-4c22-b375-ade2f0d3499f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm\" (UID: \"235a8e7d-4ca5-4c22-b375-ade2f0d3499f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm"
Jul 27 21:46:23.943296 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.943267    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5tjk\" (UniqueName: \"kubernetes.io/projected/235a8e7d-4ca5-4c22-b375-ade2f0d3499f-kube-api-access-w5tjk\") pod \"managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm\" (UID: \"235a8e7d-4ca5-4c22-b375-ade2f0d3499f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm"
Jul 27 21:46:23.943399 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.943278    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d57g6\" (UniqueName: \"kubernetes.io/projected/c70e3eca-af15-4b11-ab52-71b75ea220a4-kube-api-access-d57g6\") pod \"downloads-7c97dc4555-cpdtd\" (UID: \"c70e3eca-af15-4b11-ab52-71b75ea220a4\") " pod="openshift-console/downloads-7c97dc4555-cpdtd"
Jul 27 21:46:23.995396 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:23.995358    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"
Jul 27 21:46:24.009523 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.009471    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-cpdtd"
Jul 27 21:46:24.016758 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.016722    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm"
Jul 27 21:46:24.078552 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.077652    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hpzp8" event={"ID":"a03739ad-aa56-494b-9c4c-bd038c23af2a","Type":"ContainerStarted","Data":"a8724f884509a62d7fd3d7f879cee46fc7ed67358e23911ef2bce4a651ba64b7"}
Jul 27 21:46:24.078552 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.077721    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hpzp8" event={"ID":"a03739ad-aa56-494b-9c4c-bd038c23af2a","Type":"ContainerStarted","Data":"2737ed1f734f208caf2d8a803201cfbffe35b65ef437e96aa6cb4fa9de13b14b"}
Jul 27 21:46:24.157604 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.157353    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8f42l\""
Jul 27 21:46:24.165314 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.165272    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:46:24.180344 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.180206    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cmbrq\""
Jul 27 21:46:24.190782 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.188474    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"
Jul 27 21:46:24.190782 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.189543    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hpzp8" podStartSLOduration=66.609912073 podStartE2EDuration="1m8.18952338s" totalImagesPullingTime="1.579611307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:24.096048338 +0000 UTC m=+68.907395710" watchObservedRunningTime="2026-07-27 21:46:24.18952338 +0000 UTC m=+69.000870752"
Jul 27 21:46:24.190782 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.190762    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55"]
Jul 27 21:46:24.215490 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.214251    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-cpdtd"]
Jul 27 21:46:24.215490 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.214977    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jb57h\""
Jul 27 21:46:24.217314 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:46:24.217267    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70e3eca_af15_4b11_ab52_71b75ea220a4.slice/crio-7cc3ba98111e791065d922eca59a5071db2833da348a5a976cc5ca69c1e30255 WatchSource:0}: Error finding container 7cc3ba98111e791065d922eca59a5071db2833da348a5a976cc5ca69c1e30255: Status 404 returned error can't find the container with id 7cc3ba98111e791065d922eca59a5071db2833da348a5a976cc5ca69c1e30255
Jul 27 21:46:24.223157 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.223136    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:46:24.253039 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.252973    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm"]
Jul 27 21:46:24.261351 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:46:24.261294    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod235a8e7d_4ca5_4c22_b375_ade2f0d3499f.slice/crio-748612d559fd43991cab6fe2f693eec8a8ccc48cb0968a0435663dc9cb16edd2 WatchSource:0}: Error finding container 748612d559fd43991cab6fe2f693eec8a8ccc48cb0968a0435663dc9cb16edd2: Status 404 returned error can't find the container with id 748612d559fd43991cab6fe2f693eec8a8ccc48cb0968a0435663dc9cb16edd2
Jul 27 21:46:24.376996 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.376859    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74db6cf7db-lk5q4"]
Jul 27 21:46:24.379752 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:46:24.379722    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9593fc04_69be_49b6_b0d1_dda41059ce61.slice/crio-07cdb0f9cbd59dcabcce4828164c4849aad045c9fbc245b6e0c3197f91e51c77 WatchSource:0}: Error finding container 07cdb0f9cbd59dcabcce4828164c4849aad045c9fbc245b6e0c3197f91e51c77: Status 404 returned error can't find the container with id 07cdb0f9cbd59dcabcce4828164c4849aad045c9fbc245b6e0c3197f91e51c77
Jul 27 21:46:24.403196 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.403162    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr"]
Jul 27 21:46:24.427075 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:24.426945    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dpv6j"]
Jul 27 21:46:25.082992 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:25.082950    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55" event={"ID":"ddf729fb-893f-4607-8835-d9505ef490f4","Type":"ContainerStarted","Data":"a7a1fff148a8e10ae9348795fe2a4d413b126793ba2c9beccd986dc348c3dfe1"}
Jul 27 21:46:25.084332 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:25.084301    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4" event={"ID":"9593fc04-69be-49b6-b0d1-dda41059ce61","Type":"ContainerStarted","Data":"07cdb0f9cbd59dcabcce4828164c4849aad045c9fbc245b6e0c3197f91e51c77"}
Jul 27 21:46:25.085640 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:25.085612    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm" event={"ID":"235a8e7d-4ca5-4c22-b375-ade2f0d3499f","Type":"ContainerStarted","Data":"748612d559fd43991cab6fe2f693eec8a8ccc48cb0968a0435663dc9cb16edd2"}
Jul 27 21:46:25.086787 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:25.086764    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-cpdtd" event={"ID":"c70e3eca-af15-4b11-ab52-71b75ea220a4","Type":"ContainerStarted","Data":"7cc3ba98111e791065d922eca59a5071db2833da348a5a976cc5ca69c1e30255"}
Jul 27 21:46:25.653533 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:25.653490    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bf6dz\" (UID: \"ebc50b72-b32f-42b2-b817-09d4ed57857b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"
Jul 27 21:46:25.656389 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:25.656361    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ebc50b72-b32f-42b2-b817-09d4ed57857b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bf6dz\" (UID: \"ebc50b72-b32f-42b2-b817-09d4ed57857b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"
Jul 27 21:46:25.818400 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:25.818370    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-lhp6l\""
Jul 27 21:46:25.823998 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:46:25.823967    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5933c6_64fd_4d0d_b8f9_c32fd675d89d.slice/crio-6771065189d9a2ef4a03e46a20472dd73dc3a90db6781bb1d3e984515f6df68f WatchSource:0}: Error finding container 6771065189d9a2ef4a03e46a20472dd73dc3a90db6781bb1d3e984515f6df68f: Status 404 returned error can't find the container with id 6771065189d9a2ef4a03e46a20472dd73dc3a90db6781bb1d3e984515f6df68f
Jul 27 21:46:25.825667 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:25.825551    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"
Jul 27 21:46:26.085606 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:26.085575    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz"]
Jul 27 21:46:26.093789 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:46:26.093700    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc50b72_b32f_42b2_b817_09d4ed57857b.slice/crio-aae0c6b2f1892b82dc7bb680dcced53e3c2b9edaa0dea0cfb6f87be3e17ac3eb WatchSource:0}: Error finding container aae0c6b2f1892b82dc7bb680dcced53e3c2b9edaa0dea0cfb6f87be3e17ac3eb: Status 404 returned error can't find the container with id aae0c6b2f1892b82dc7bb680dcced53e3c2b9edaa0dea0cfb6f87be3e17ac3eb
Jul 27 21:46:26.109435 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:26.109299    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dpv6j" event={"ID":"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d","Type":"ContainerStarted","Data":"6771065189d9a2ef4a03e46a20472dd73dc3a90db6781bb1d3e984515f6df68f"}
Jul 27 21:46:26.125921 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:26.125877    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr" event={"ID":"b7ec81ff-7926-4a88-89c3-ab110245f6e7","Type":"ContainerStarted","Data":"0fb2e23f424c4ca6b19faeba4bb5f89cc38a7b6490be17084900ebd34e86ca72"}
Jul 27 21:46:26.129734 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:26.129635    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4" event={"ID":"9593fc04-69be-49b6-b0d1-dda41059ce61","Type":"ContainerStarted","Data":"236bb405c57c9b74905a58dbd977ca8ab8a37ccd72ec9d020308206227dc2ca3"}
Jul 27 21:46:26.130047 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:26.129891    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:46:27.147345 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:27.147306    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8tk6w" event={"ID":"4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2","Type":"ContainerStarted","Data":"97376ebcf7f48893a224f60403cc65e8b951ea8985cf08b8faf290cbfa9b0103"}
Jul 27 21:46:27.151661 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:27.151628    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz" event={"ID":"ebc50b72-b32f-42b2-b817-09d4ed57857b","Type":"ContainerStarted","Data":"aae0c6b2f1892b82dc7bb680dcced53e3c2b9edaa0dea0cfb6f87be3e17ac3eb"}
Jul 27 21:46:27.164577 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:27.164515    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4" podStartSLOduration=71.164489712 podStartE2EDuration="1m11.164489712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:26.17554815 +0000 UTC m=+70.986895523" watchObservedRunningTime="2026-07-27 21:46:27.164489712 +0000 UTC m=+71.975837085"
Jul 27 21:46:29.549039 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.548963    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8tk6w" podStartSLOduration=69.307806889 podStartE2EDuration="1m13.548937975s" totalImagesPullingTime="4.241131086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:27.165623398 +0000 UTC m=+71.976970771" watchObservedRunningTime="2026-07-27 21:46:29.548937975 +0000 UTC m=+74.360285331"
Jul 27 21:46:29.549497 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.549403    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x7lth"]
Jul 27 21:46:29.552965 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.552934    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:29.555359 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.555333    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 21:46:29.555639 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.555621    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 21:46:29.556241 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.556194    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 21:46:29.556514 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.556480    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 21:46:29.556996 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.556978    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7smr8\""
Jul 27 21:46:29.560593 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.560570    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x7lth"]
Jul 27 21:46:29.689114 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.689070    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:29.689318 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.689130    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:29.689318 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.689160    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4ckx\" (UniqueName: \"kubernetes.io/projected/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-api-access-p4ckx\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:29.689318 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.689195    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-crio-socket\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:29.689318 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.689243    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-data-volume\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:29.790370 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.790332    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:29.790370 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.790387    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:29.790614 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.790410    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4ckx\" (UniqueName: \"kubernetes.io/projected/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-api-access-p4ckx\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:29.790614 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.790443    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-crio-socket\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:29.790614 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.790463    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-data-volume\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:29.790872 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.790752    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-data-volume\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:29.790872 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.790845    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-crio-socket\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:29.790872 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:29.790858    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:46:29.791102 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:29.790894    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:29.791102 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:29.790956    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm podName:dea6eb9c-7849-4fe0-9a80-d6f867b98ea1 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:30.290924869 +0000 UTC m=+75.102272222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x7lth" (UID: "dea6eb9c-7849-4fe0-9a80-d6f867b98ea1") : configmap "kube-rbac-proxy" not found
Jul 27 21:46:29.791102 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:29.790977    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls podName:dea6eb9c-7849-4fe0-9a80-d6f867b98ea1 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:30.290967607 +0000 UTC m=+75.102314963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7lth" (UID: "dea6eb9c-7849-4fe0-9a80-d6f867b98ea1") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:29.805299 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:29.800130    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4ckx\" (UniqueName: \"kubernetes.io/projected/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-api-access-p4ckx\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:30.296069 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:30.296034    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:30.296270 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:30.296093    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:30.296270 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:30.296204    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:46:30.296270 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:30.296249    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:30.296410 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:30.296286    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm podName:dea6eb9c-7849-4fe0-9a80-d6f867b98ea1 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:31.296266824 +0000 UTC m=+76.107614178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x7lth" (UID: "dea6eb9c-7849-4fe0-9a80-d6f867b98ea1") : configmap "kube-rbac-proxy" not found
Jul 27 21:46:30.296410 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:30.296319    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls podName:dea6eb9c-7849-4fe0-9a80-d6f867b98ea1 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:31.296300495 +0000 UTC m=+76.107647850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7lth" (UID: "dea6eb9c-7849-4fe0-9a80-d6f867b98ea1") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:31.305188 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:31.305143    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:31.305594 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:31.305206    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:31.305594 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:31.305327    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:46:31.305594 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:31.305349    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:31.305594 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:31.305390    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm podName:dea6eb9c-7849-4fe0-9a80-d6f867b98ea1 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:33.305371491 +0000 UTC m=+78.116719037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x7lth" (UID: "dea6eb9c-7849-4fe0-9a80-d6f867b98ea1") : configmap "kube-rbac-proxy" not found
Jul 27 21:46:31.305594 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:31.305423    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls podName:dea6eb9c-7849-4fe0-9a80-d6f867b98ea1 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:33.305405013 +0000 UTC m=+78.116752368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7lth" (UID: "dea6eb9c-7849-4fe0-9a80-d6f867b98ea1") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:31.570057 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:31.569980    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fk2vd_519f6723-a7eb-4bbd-82db-e6664825df3c/dns-node-resolver/0.log"
Jul 27 21:46:31.970733 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:31.970700    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-74db6cf7db-lk5q4_9593fc04-69be-49b6-b0d1-dda41059ce61/registry/0.log"
Jul 27 21:46:32.573099 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:32.573060    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-75kmz_135bb451-1678-49a0-b5bb-33e63a8ac0c2/node-ca/0.log"
Jul 27 21:46:33.184583 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:33.184518    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr" event={"ID":"b7ec81ff-7926-4a88-89c3-ab110245f6e7","Type":"ContainerStarted","Data":"4df7f03a703b11d6918a8a69c2836e6d4f72c38aebe2ef0ef1e6ad73363be471"}
Jul 27 21:46:33.186482 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:33.186430    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm" event={"ID":"235a8e7d-4ca5-4c22-b375-ade2f0d3499f","Type":"ContainerStarted","Data":"afb76566f86359cc1345eb0b689036897ac9b301621e56cbc7a03190b7ba0a68"}
Jul 27 21:46:33.326911 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:33.326877    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:33.327066 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:33.326926    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:33.327673 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:33.327353    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:46:33.327673 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:33.327431    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm podName:dea6eb9c-7849-4fe0-9a80-d6f867b98ea1 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:37.327409018 +0000 UTC m=+82.138756383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x7lth" (UID: "dea6eb9c-7849-4fe0-9a80-d6f867b98ea1") : configmap "kube-rbac-proxy" not found
Jul 27 21:46:33.327673 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:33.327529    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:33.327673 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:33.327570    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls podName:dea6eb9c-7849-4fe0-9a80-d6f867b98ea1 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:37.327557842 +0000 UTC m=+82.138905192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7lth" (UID: "dea6eb9c-7849-4fe0-9a80-d6f867b98ea1") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:33.374235 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:33.374180    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bfc7z_f2576233-1733-4b8a-9056-09acb732b3d0/serve-healthcheck-canary/0.log"
Jul 27 21:46:34.196129 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:34.195631    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr" event={"ID":"b7ec81ff-7926-4a88-89c3-ab110245f6e7","Type":"ContainerStarted","Data":"98c5f4f092f52c55a3171db2fdf3c86044a11b8faad032ccf181b7dafbe1d4b9"}
Jul 27 21:46:34.197288 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:34.197254    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz" event={"ID":"ebc50b72-b32f-42b2-b817-09d4ed57857b","Type":"ContainerStarted","Data":"a03d6b2eaf2ebf02d866d91e55a84297af07d77e2927dab601489d0ef71dbd2f"}
Jul 27 21:46:34.199456 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:34.199432    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55" event={"ID":"ddf729fb-893f-4607-8835-d9505ef490f4","Type":"ContainerStarted","Data":"0c6adfbf1e3cca02a72ba36686d2d98b14c441539a39cffc28fd5a6a4917ce37"}
Jul 27 21:46:34.199592 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:34.199472    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55" event={"ID":"ddf729fb-893f-4607-8835-d9505ef490f4","Type":"ContainerStarted","Data":"9388c2188fc876f2a75f98247973cb4c996c1001a9cba1857bf1604d28729c49"}
Jul 27 21:46:34.199592 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:34.199490    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55" event={"ID":"ddf729fb-893f-4607-8835-d9505ef490f4","Type":"ContainerStarted","Data":"1ed18fef8a1f93af73a846605d51432cf76cc7e67f2a6b1ad6545b2816b9b220"}
Jul 27 21:46:34.201473 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:34.201444    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dpv6j" event={"ID":"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d","Type":"ContainerStarted","Data":"8d33b22b62133bcfb06d26c3f80a8d445adfce2d6f660c4fc4ef08166949e2a3"}
Jul 27 21:46:34.201584 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:34.201495    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dpv6j" event={"ID":"2c5933c6-64fd-4d0d-b8f9-c32fd675d89d","Type":"ContainerStarted","Data":"4b5714e1028d30cd4ab0c8a82ae9e6ce59b40be2ea1c263265ac1af350bae367"}
Jul 27 21:46:34.213187 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:34.213140    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dc7f6cfd4-f4lsm" podStartSLOduration=2.51045132 podStartE2EDuration="11.213124841s" totalImagesPullingTime="8.702673521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:33.213524953 +0000 UTC m=+78.024872326" watchObservedRunningTime="2026-07-27 21:46:34.213124841 +0000 UTC m=+79.024472191"
Jul 27 21:46:34.213502 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:34.213459    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4bnsr" podStartSLOduration=70.136909027 podStartE2EDuration="1m17.21344832s" totalImagesPullingTime="7.076539293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:34.212624604 +0000 UTC m=+79.023971976" watchObservedRunningTime="2026-07-27 21:46:34.21344832 +0000 UTC m=+79.024795692"
Jul 27 21:46:34.234066 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:34.234016    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f76d679b9-x2q55" podStartSLOduration=2.437491329 podStartE2EDuration="11.234000739s" totalImagesPullingTime="8.79650941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:34.233197969 +0000 UTC m=+79.044545352" watchObservedRunningTime="2026-07-27 21:46:34.234000739 +0000 UTC m=+79.045348138"
Jul 27 21:46:34.271174 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:34.271125    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dpv6j" podStartSLOduration=35.123748495 podStartE2EDuration="42.271108322s" totalImagesPullingTime="7.147359827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:34.270787649 +0000 UTC m=+79.082135024" watchObservedRunningTime="2026-07-27 21:46:34.271108322 +0000 UTC m=+79.082455695"
Jul 27 21:46:34.271588 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:34.271557    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bf6dz" podStartSLOduration=34.402885893 podStartE2EDuration="41.271545776s" totalImagesPullingTime="6.868659883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:34.249529211 +0000 UTC m=+79.060876585" watchObservedRunningTime="2026-07-27 21:46:34.271545776 +0000 UTC m=+79.082893155"
Jul 27 21:46:35.206426 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:35.206399    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:46:37.364161 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.364004    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:37.364161 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.364123    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:37.364783 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:37.364156    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:46:37.364783 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:37.364270    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:37.364783 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:37.364273    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm podName:dea6eb9c-7849-4fe0-9a80-d6f867b98ea1 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:45.364248979 +0000 UTC m=+90.175596347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x7lth" (UID: "dea6eb9c-7849-4fe0-9a80-d6f867b98ea1") : configmap "kube-rbac-proxy" not found
Jul 27 21:46:37.364783 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:37.364345    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls podName:dea6eb9c-7849-4fe0-9a80-d6f867b98ea1 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:45.364329507 +0000 UTC m=+90.175676869 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7lth" (UID: "dea6eb9c-7849-4fe0-9a80-d6f867b98ea1") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:37.769800 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.769721    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rq46q"]
Jul 27 21:46:37.772348 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.772322    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.775351 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.775307    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 21:46:37.775509 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.775407    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 21:46:37.775793 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.775771    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 21:46:37.776960 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.776940    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 21:46:37.776960 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.776951    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xtsgm\""
Jul 27 21:46:37.777104 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.776945    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 21:46:37.777104 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.776984    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 21:46:37.868946 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.868906    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16c022ac-d280-47d5-a150-76485eaec3f9-metrics-client-ca\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.868946 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.868959    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16c022ac-d280-47d5-a150-76485eaec3f9-root\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.869208 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.869028    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-wtmp\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.869208 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.869107    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.869208 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.869144    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2lsk\" (UniqueName: \"kubernetes.io/projected/16c022ac-d280-47d5-a150-76485eaec3f9-kube-api-access-m2lsk\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.869208 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.869174    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16c022ac-d280-47d5-a150-76485eaec3f9-sys\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.869451 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.869323    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-textfile\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.869451 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.869375    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.869451 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.869424    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-tls\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.970893 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.970857    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-textfile\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.971120 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.970928    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.971120 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.970965    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-tls\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.971120 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.971029    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16c022ac-d280-47d5-a150-76485eaec3f9-metrics-client-ca\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.971120 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.971056    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16c022ac-d280-47d5-a150-76485eaec3f9-root\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.971120 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.971079    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-wtmp\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.971338 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.971130    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.971338 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.971159    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2lsk\" (UniqueName: \"kubernetes.io/projected/16c022ac-d280-47d5-a150-76485eaec3f9-kube-api-access-m2lsk\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.971338 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.971184    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16c022ac-d280-47d5-a150-76485eaec3f9-sys\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.971338 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.971288    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-textfile\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.971338 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.971309    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16c022ac-d280-47d5-a150-76485eaec3f9-sys\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.971508 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:37.971431    2549 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 21:46:37.971508 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.971441    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-wtmp\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.971508 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.971478    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16c022ac-d280-47d5-a150-76485eaec3f9-root\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.971508 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:46:37.971494    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-tls podName:16c022ac-d280-47d5-a150-76485eaec3f9 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:38.471475592 +0000 UTC m=+83.282822959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-tls") pod "node-exporter-rq46q" (UID: "16c022ac-d280-47d5-a150-76485eaec3f9") : secret "node-exporter-tls" not found
Jul 27 21:46:37.971646 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.971515    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.972028 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.971980    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16c022ac-d280-47d5-a150-76485eaec3f9-metrics-client-ca\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.974366 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.974338    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:37.982441 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:37.982182    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2lsk\" (UniqueName: \"kubernetes.io/projected/16c022ac-d280-47d5-a150-76485eaec3f9-kube-api-access-m2lsk\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:38.476744 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:38.476710    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-tls\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:38.479278 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:38.479251    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16c022ac-d280-47d5-a150-76485eaec3f9-node-exporter-tls\") pod \"node-exporter-rq46q\" (UID: \"16c022ac-d280-47d5-a150-76485eaec3f9\") " pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:38.684636 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:38.684604    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rq46q"
Jul 27 21:46:43.613554 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:46:43.613522    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c022ac_d280_47d5_a150_76485eaec3f9.slice/crio-a3f31d9dae377de3b3a6dd20a2e33eb052c134e2ec55abbd8d7a456e4567787f WatchSource:0}: Error finding container a3f31d9dae377de3b3a6dd20a2e33eb052c134e2ec55abbd8d7a456e4567787f: Status 404 returned error can't find the container with id a3f31d9dae377de3b3a6dd20a2e33eb052c134e2ec55abbd8d7a456e4567787f
Jul 27 21:46:44.244597 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:44.244553    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-cpdtd" event={"ID":"c70e3eca-af15-4b11-ab52-71b75ea220a4","Type":"ContainerStarted","Data":"e6750697e32a83ee15efb5df039a214780d5b0be951aa93572be8e806d61ac8d"}
Jul 27 21:46:44.245692 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:44.245663    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-cpdtd"
Jul 27 21:46:44.247964 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:44.247925    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rq46q" event={"ID":"16c022ac-d280-47d5-a150-76485eaec3f9","Type":"ContainerStarted","Data":"a3f31d9dae377de3b3a6dd20a2e33eb052c134e2ec55abbd8d7a456e4567787f"}
Jul 27 21:46:44.258740 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:44.258706    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-cpdtd"
Jul 27 21:46:44.269845 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:44.268497    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-cpdtd" podStartSLOduration=1.700396998 podStartE2EDuration="21.26847616s" totalImagesPullingTime="19.568079162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:44.265712511 +0000 UTC m=+89.077059882" watchObservedRunningTime="2026-07-27 21:46:44.26847616 +0000 UTC m=+89.079823532"
Jul 27 21:46:45.212517 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:45.212487    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dpv6j"
Jul 27 21:46:45.254245 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:45.254177    2549 generic.go:356] "Generic (PLEG): container finished" podID="16c022ac-d280-47d5-a150-76485eaec3f9" containerID="63de2068e5a471c7eeaafaf3fcaae3c80b84e610b28842f33d7691a60e33db36" exitCode=0
Jul 27 21:46:45.254871 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:45.254835    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rq46q" event={"ID":"16c022ac-d280-47d5-a150-76485eaec3f9","Type":"ContainerDied","Data":"63de2068e5a471c7eeaafaf3fcaae3c80b84e610b28842f33d7691a60e33db36"}
Jul 27 21:46:45.440938 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:45.440899    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:45.441100 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:45.440970    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:45.441623 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:45.441594    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:45.443812 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:45.443791    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dea6eb9c-7849-4fe0-9a80-d6f867b98ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7lth\" (UID: \"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1\") " pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:45.467335 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:45.467246    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x7lth"
Jul 27 21:46:45.678677 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:45.678638    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x7lth"]
Jul 27 21:46:45.686609 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:46:45.686573    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea6eb9c_7849_4fe0_9a80_d6f867b98ea1.slice/crio-d54d1ce69daa206e5678d5df8a18c5dd64ffdc6ba615c4c947db7fc1869dfa73 WatchSource:0}: Error finding container d54d1ce69daa206e5678d5df8a18c5dd64ffdc6ba615c4c947db7fc1869dfa73: Status 404 returned error can't find the container with id d54d1ce69daa206e5678d5df8a18c5dd64ffdc6ba615c4c947db7fc1869dfa73
Jul 27 21:46:46.169392 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:46.169355    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74db6cf7db-lk5q4"]
Jul 27 21:46:46.175812 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:46.175755    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:46:46.267543 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:46.267483    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7lth" event={"ID":"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1","Type":"ContainerStarted","Data":"d54d1ce69daa206e5678d5df8a18c5dd64ffdc6ba615c4c947db7fc1869dfa73"}
Jul 27 21:46:46.274252 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:46.274120    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rq46q" event={"ID":"16c022ac-d280-47d5-a150-76485eaec3f9","Type":"ContainerStarted","Data":"1b375ce94c479145e792822fd1f5e90c4038c14f02c4d7608caad178c87795f6"}
Jul 27 21:46:46.274252 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:46.274194    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rq46q" event={"ID":"16c022ac-d280-47d5-a150-76485eaec3f9","Type":"ContainerStarted","Data":"eb727e101e55e62a21fc233ee8e1b07bd96bc3dccc0991641d3e4bd5261a2733"}
Jul 27 21:46:46.296834 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:46.296773    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rq46q" podStartSLOduration=8.143441177 podStartE2EDuration="9.296747107s" totalImagesPullingTime="1.066376328s" totalInitContainerRuntime="86.929602ms" isStatefulPod=true podCreationTimestamp="2026-07-27 21:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:46.296408106 +0000 UTC m=+91.107755479" watchObservedRunningTime="2026-07-27 21:46:46.296747107 +0000 UTC m=+91.108094480"
Jul 27 21:46:53.074522 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:53.074487    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fqsbg"
Jul 27 21:46:53.312115 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:53.312074    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7lth" event={"ID":"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1","Type":"ContainerStarted","Data":"1e106a8184a13777ee88d57fa230d3232e8b94fe59fb865ed63bf9139cd8cd94"}
Jul 27 21:46:58.335597 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:46:58.335560    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7lth" event={"ID":"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1","Type":"ContainerStarted","Data":"1669e8a986d67f858b3506fb53cebcb0b677311644cb0c7758ecdfef56ae7ecb"}
Jul 27 21:47:02.354612 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:02.354571    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7lth" event={"ID":"dea6eb9c-7849-4fe0-9a80-d6f867b98ea1","Type":"ContainerStarted","Data":"30e8ea8a8bd6c3a8e29fb91db31e0f2a1cdc83ee7e0afd35e85c7c435165625c"}
Jul 27 21:47:02.373321 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:02.373267    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x7lth" podStartSLOduration=17.795455637 podStartE2EDuration="33.373250783s" totalImagesPullingTime="15.577795146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:46:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:47:02.373085091 +0000 UTC m=+107.184432463" watchObservedRunningTime="2026-07-27 21:47:02.373250783 +0000 UTC m=+107.184598154"
Jul 27 21:47:11.197861 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.197778    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4" podUID="9593fc04-69be-49b6-b0d1-dda41059ce61" containerName="registry" containerID="cri-o://236bb405c57c9b74905a58dbd977ca8ab8a37ccd72ec9d020308206227dc2ca3" gracePeriod=30
Jul 27 21:47:11.389082 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.389040    2549 generic.go:356] "Generic (PLEG): container finished" podID="9593fc04-69be-49b6-b0d1-dda41059ce61" containerID="236bb405c57c9b74905a58dbd977ca8ab8a37ccd72ec9d020308206227dc2ca3" exitCode=0
Jul 27 21:47:11.389259 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.389111    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4" event={"ID":"9593fc04-69be-49b6-b0d1-dda41059ce61","Type":"ContainerDied","Data":"236bb405c57c9b74905a58dbd977ca8ab8a37ccd72ec9d020308206227dc2ca3"}
Jul 27 21:47:11.478185 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.478103    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:47:11.569438 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.569389    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-certificates\") pod \"9593fc04-69be-49b6-b0d1-dda41059ce61\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") "
Jul 27 21:47:11.569438 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.569443    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9593fc04-69be-49b6-b0d1-dda41059ce61-trusted-ca\") pod \"9593fc04-69be-49b6-b0d1-dda41059ce61\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") "
Jul 27 21:47:11.569703 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.569486    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls\") pod \"9593fc04-69be-49b6-b0d1-dda41059ce61\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") "
Jul 27 21:47:11.569703 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.569506    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74h2c\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-kube-api-access-74h2c\") pod \"9593fc04-69be-49b6-b0d1-dda41059ce61\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") "
Jul 27 21:47:11.569703 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.569528    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9593fc04-69be-49b6-b0d1-dda41059ce61-ca-trust-extracted\") pod \"9593fc04-69be-49b6-b0d1-dda41059ce61\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") "
Jul 27 21:47:11.569703 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.569568    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9593fc04-69be-49b6-b0d1-dda41059ce61-image-registry-private-configuration\") pod \"9593fc04-69be-49b6-b0d1-dda41059ce61\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") "
Jul 27 21:47:11.569703 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.569600    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9593fc04-69be-49b6-b0d1-dda41059ce61-installation-pull-secrets\") pod \"9593fc04-69be-49b6-b0d1-dda41059ce61\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") "
Jul 27 21:47:11.569703 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.569625    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-bound-sa-token\") pod \"9593fc04-69be-49b6-b0d1-dda41059ce61\" (UID: \"9593fc04-69be-49b6-b0d1-dda41059ce61\") "
Jul 27 21:47:11.569979 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.569953    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9593fc04-69be-49b6-b0d1-dda41059ce61" (UID: "9593fc04-69be-49b6-b0d1-dda41059ce61"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:47:11.570038 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.570010    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9593fc04-69be-49b6-b0d1-dda41059ce61-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9593fc04-69be-49b6-b0d1-dda41059ce61" (UID: "9593fc04-69be-49b6-b0d1-dda41059ce61"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:47:11.572271 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.572191    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9593fc04-69be-49b6-b0d1-dda41059ce61-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9593fc04-69be-49b6-b0d1-dda41059ce61" (UID: "9593fc04-69be-49b6-b0d1-dda41059ce61"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:47:11.572271 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.572202    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-kube-api-access-74h2c" (OuterVolumeSpecName: "kube-api-access-74h2c") pod "9593fc04-69be-49b6-b0d1-dda41059ce61" (UID: "9593fc04-69be-49b6-b0d1-dda41059ce61"). InnerVolumeSpecName "kube-api-access-74h2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:47:11.572566 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.572542    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9593fc04-69be-49b6-b0d1-dda41059ce61" (UID: "9593fc04-69be-49b6-b0d1-dda41059ce61"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:47:11.572635 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.572561    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9593fc04-69be-49b6-b0d1-dda41059ce61-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9593fc04-69be-49b6-b0d1-dda41059ce61" (UID: "9593fc04-69be-49b6-b0d1-dda41059ce61"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:47:11.572676 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.572634    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9593fc04-69be-49b6-b0d1-dda41059ce61" (UID: "9593fc04-69be-49b6-b0d1-dda41059ce61"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:47:11.578609 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.578579    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9593fc04-69be-49b6-b0d1-dda41059ce61-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9593fc04-69be-49b6-b0d1-dda41059ce61" (UID: "9593fc04-69be-49b6-b0d1-dda41059ce61"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:47:11.671076 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.671029    2549 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-certificates\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:47:11.671076 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.671070    2549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9593fc04-69be-49b6-b0d1-dda41059ce61-trusted-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:47:11.671076 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.671079    2549 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-registry-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:47:11.671076 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.671087    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74h2c\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-kube-api-access-74h2c\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:47:11.671380 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.671095    2549 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9593fc04-69be-49b6-b0d1-dda41059ce61-ca-trust-extracted\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:47:11.671380 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.671105    2549 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9593fc04-69be-49b6-b0d1-dda41059ce61-image-registry-private-configuration\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:47:11.671380 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.671115    2549 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9593fc04-69be-49b6-b0d1-dda41059ce61-installation-pull-secrets\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:47:11.671380 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:11.671124    2549 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9593fc04-69be-49b6-b0d1-dda41059ce61-bound-sa-token\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:47:12.394493 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:12.394468    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4"
Jul 27 21:47:12.394493 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:12.394479    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74db6cf7db-lk5q4" event={"ID":"9593fc04-69be-49b6-b0d1-dda41059ce61","Type":"ContainerDied","Data":"07cdb0f9cbd59dcabcce4828164c4849aad045c9fbc245b6e0c3197f91e51c77"}
Jul 27 21:47:12.394956 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:12.394535    2549 scope.go:117] "RemoveContainer" containerID="236bb405c57c9b74905a58dbd977ca8ab8a37ccd72ec9d020308206227dc2ca3"
Jul 27 21:47:12.413834 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:12.413805    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74db6cf7db-lk5q4"]
Jul 27 21:47:12.420901 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:12.417694    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74db6cf7db-lk5q4"]
Jul 27 21:47:13.698876 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:13.698841    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9593fc04-69be-49b6-b0d1-dda41059ce61" path="/var/lib/kubelet/pods/9593fc04-69be-49b6-b0d1-dda41059ce61/volumes"
Jul 27 21:47:15.406411 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:15.406373    2549 generic.go:356] "Generic (PLEG): container finished" podID="5c7b3d47-576c-4657-a558-69c78baeebad" containerID="8c70f3cc158d61b0bb7381cffd48a45cb928cc6b8ea1bdaed642a373d9d629eb" exitCode=0
Jul 27 21:47:15.406796 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:15.406443    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc" event={"ID":"5c7b3d47-576c-4657-a558-69c78baeebad","Type":"ContainerDied","Data":"8c70f3cc158d61b0bb7381cffd48a45cb928cc6b8ea1bdaed642a373d9d629eb"}
Jul 27 21:47:15.406796 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:15.406760    2549 scope.go:117] "RemoveContainer" containerID="8c70f3cc158d61b0bb7381cffd48a45cb928cc6b8ea1bdaed642a373d9d629eb"
Jul 27 21:47:16.411761 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:16.411726    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc" event={"ID":"5c7b3d47-576c-4657-a558-69c78baeebad","Type":"ContainerStarted","Data":"046d0bed624bb5d7243d74e32235416f9ca97b0f1ff7b79890243162130cac67"}
Jul 27 21:47:31.473128 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:31.473092    2549 generic.go:356] "Generic (PLEG): container finished" podID="67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc" containerID="08eac3f55712e81cfbfa66ddbc44e750e9f37aff88807b754a1bdf5fd442e3eb" exitCode=0
Jul 27 21:47:31.473533 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:31.473135    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl" event={"ID":"67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc","Type":"ContainerDied","Data":"08eac3f55712e81cfbfa66ddbc44e750e9f37aff88807b754a1bdf5fd442e3eb"}
Jul 27 21:47:31.473533 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:31.473517    2549 scope.go:117] "RemoveContainer" containerID="08eac3f55712e81cfbfa66ddbc44e750e9f37aff88807b754a1bdf5fd442e3eb"
Jul 27 21:47:32.478546 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:47:32.478514    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ptvcl" event={"ID":"67023c0a-97ab-4d2e-ae5c-c7ff5aae7ecc","Type":"ContainerStarted","Data":"7e56c920b524eda5368f5ed89961e8cdb4f118f0b25b91d523f925d8a6f36e96"}
Jul 27 21:49:26.669520 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.669485    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j6sf2"]
Jul 27 21:49:26.670033 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.669739    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9593fc04-69be-49b6-b0d1-dda41059ce61" containerName="registry"
Jul 27 21:49:26.670033 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.669750    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="9593fc04-69be-49b6-b0d1-dda41059ce61" containerName="registry"
Jul 27 21:49:26.670033 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.669805    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="9593fc04-69be-49b6-b0d1-dda41059ce61" containerName="registry"
Jul 27 21:49:26.671573 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.671559    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j6sf2"
Jul 27 21:49:26.676885 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.676856    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c29f149-e54a-41b5-accb-f1a848e0e2aa-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j6sf2\" (UID: \"0c29f149-e54a-41b5-accb-f1a848e0e2aa\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j6sf2"
Jul 27 21:49:26.677026 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.676917    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dljj6\" (UniqueName: \"kubernetes.io/projected/0c29f149-e54a-41b5-accb-f1a848e0e2aa-kube-api-access-dljj6\") pod \"openshift-lws-operator-fd8ccff4c-j6sf2\" (UID: \"0c29f149-e54a-41b5-accb-f1a848e0e2aa\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j6sf2"
Jul 27 21:49:26.678773 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.678754    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jhs9k\""
Jul 27 21:49:26.679254 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.679240    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:49:26.680183 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.680166    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 21:49:26.688767 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.688738    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j6sf2"]
Jul 27 21:49:26.777248 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.777192    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c29f149-e54a-41b5-accb-f1a848e0e2aa-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j6sf2\" (UID: \"0c29f149-e54a-41b5-accb-f1a848e0e2aa\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j6sf2"
Jul 27 21:49:26.777411 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.777279    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dljj6\" (UniqueName: \"kubernetes.io/projected/0c29f149-e54a-41b5-accb-f1a848e0e2aa-kube-api-access-dljj6\") pod \"openshift-lws-operator-fd8ccff4c-j6sf2\" (UID: \"0c29f149-e54a-41b5-accb-f1a848e0e2aa\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j6sf2"
Jul 27 21:49:26.777574 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.777554    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c29f149-e54a-41b5-accb-f1a848e0e2aa-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j6sf2\" (UID: \"0c29f149-e54a-41b5-accb-f1a848e0e2aa\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j6sf2"
Jul 27 21:49:26.789151 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.789118    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dljj6\" (UniqueName: \"kubernetes.io/projected/0c29f149-e54a-41b5-accb-f1a848e0e2aa-kube-api-access-dljj6\") pod \"openshift-lws-operator-fd8ccff4c-j6sf2\" (UID: \"0c29f149-e54a-41b5-accb-f1a848e0e2aa\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j6sf2"
Jul 27 21:49:26.980505 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:26.980401    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j6sf2"
Jul 27 21:49:27.124771 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:27.124728    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j6sf2"]
Jul 27 21:49:27.127823 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:49:27.127796    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c29f149_e54a_41b5_accb_f1a848e0e2aa.slice/crio-26e7f2c3487b3b3a2949648a0011dd08a66d53d4da888728381def9476711427 WatchSource:0}: Error finding container 26e7f2c3487b3b3a2949648a0011dd08a66d53d4da888728381def9476711427: Status 404 returned error can't find the container with id 26e7f2c3487b3b3a2949648a0011dd08a66d53d4da888728381def9476711427
Jul 27 21:49:27.893847 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:27.893800    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j6sf2" event={"ID":"0c29f149-e54a-41b5-accb-f1a848e0e2aa","Type":"ContainerStarted","Data":"26e7f2c3487b3b3a2949648a0011dd08a66d53d4da888728381def9476711427"}
Jul 27 21:49:29.903014 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:29.902979    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j6sf2" event={"ID":"0c29f149-e54a-41b5-accb-f1a848e0e2aa","Type":"ContainerStarted","Data":"a8f372698997119964a39ace89fb8244654585796baefc31e538337227ea8af5"}
Jul 27 21:49:29.926518 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:29.926468    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j6sf2" podStartSLOduration=1.55634161 podStartE2EDuration="3.926449894s" totalImagesPullingTime="2.370108284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:49:29.924329396 +0000 UTC m=+254.735676768" watchObservedRunningTime="2026-07-27 21:49:29.926449894 +0000 UTC m=+254.737797265"
Jul 27 21:49:45.820806 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:45.820729    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"]
Jul 27 21:49:45.823001 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:45.822985    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:45.826678 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:45.826638    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jbjdz\""
Jul 27 21:49:45.826967 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:45.826950    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 21:49:45.827091 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:45.826978    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 21:49:45.827208 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:45.827193    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 21:49:45.828196 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:45.828176    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 21:49:45.866578 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:45.866546    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"]
Jul 27 21:49:45.920956 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:45.920923    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/910b8b4c-ff67-44dc-8549-c10011badb60-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:45.921130 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:45.920974    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvdzs\" (UniqueName: \"kubernetes.io/projected/910b8b4c-ff67-44dc-8549-c10011badb60-kube-api-access-jvdzs\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:45.921130 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:45.920993    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/910b8b4c-ff67-44dc-8549-c10011badb60-manifests\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:45.921130 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:45.921070    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/910b8b4c-ff67-44dc-8549-c10011badb60-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:45.921130 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:45.921100    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/910b8b4c-ff67-44dc-8549-c10011badb60-tmp\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:46.021763 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:46.021708    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/910b8b4c-ff67-44dc-8549-c10011badb60-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:46.021763 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:46.021762    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/910b8b4c-ff67-44dc-8549-c10011badb60-tmp\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:46.022001 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:46.021799    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/910b8b4c-ff67-44dc-8549-c10011badb60-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:46.022001 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:46.021836    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvdzs\" (UniqueName: \"kubernetes.io/projected/910b8b4c-ff67-44dc-8549-c10011badb60-kube-api-access-jvdzs\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:46.022001 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:46.021861    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/910b8b4c-ff67-44dc-8549-c10011badb60-manifests\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:46.022324 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:46.022300    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/910b8b4c-ff67-44dc-8549-c10011badb60-manifests\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:46.024428 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:46.024403    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/910b8b4c-ff67-44dc-8549-c10011badb60-tmp\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:46.024553 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:46.024510    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/910b8b4c-ff67-44dc-8549-c10011badb60-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:46.024553 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:46.024536    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/910b8b4c-ff67-44dc-8549-c10011badb60-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:46.049914 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:46.049886    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvdzs\" (UniqueName: \"kubernetes.io/projected/910b8b4c-ff67-44dc-8549-c10011badb60-kube-api-access-jvdzs\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7\" (UID: \"910b8b4c-ff67-44dc-8549-c10011badb60\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:46.133032 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:46.132998    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:46.285653 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:46.285619    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"]
Jul 27 21:49:46.289231 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:49:46.289194    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910b8b4c_ff67_44dc_8549_c10011badb60.slice/crio-4122523fc88f319e8f8bdca55382c6da1f47cb1134061601f8890bd833511091 WatchSource:0}: Error finding container 4122523fc88f319e8f8bdca55382c6da1f47cb1134061601f8890bd833511091: Status 404 returned error can't find the container with id 4122523fc88f319e8f8bdca55382c6da1f47cb1134061601f8890bd833511091
Jul 27 21:49:46.965876 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:46.965830    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7" event={"ID":"910b8b4c-ff67-44dc-8549-c10011badb60","Type":"ContainerStarted","Data":"4122523fc88f319e8f8bdca55382c6da1f47cb1134061601f8890bd833511091"}
Jul 27 21:49:49.980704 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:49.980671    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7" event={"ID":"910b8b4c-ff67-44dc-8549-c10011badb60","Type":"ContainerStarted","Data":"2afdaab0ca7a831c7e6fa438ce68964b067c47ea2ebdac851816bea10b274cda"}
Jul 27 21:49:50.986572 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:50.986541    2549 generic.go:356] "Generic (PLEG): container finished" podID="910b8b4c-ff67-44dc-8549-c10011badb60" containerID="2afdaab0ca7a831c7e6fa438ce68964b067c47ea2ebdac851816bea10b274cda" exitCode=0
Jul 27 21:49:50.987014 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:50.986606    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7" event={"ID":"910b8b4c-ff67-44dc-8549-c10011badb60","Type":"ContainerDied","Data":"2afdaab0ca7a831c7e6fa438ce68964b067c47ea2ebdac851816bea10b274cda"}
Jul 27 21:49:51.993149 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:51.993110    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7" event={"ID":"910b8b4c-ff67-44dc-8549-c10011badb60","Type":"ContainerStarted","Data":"feb8ce0191e1e131e8afc743214924be62781a2cd786e837da6b383caa3b4bb8"}
Jul 27 21:49:51.993652 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:51.993254    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:49:52.030957 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:49:52.030886    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7" podStartSLOduration=2.885178135 podStartE2EDuration="7.030863943s" totalImagesPullingTime="3.110705468s" totalInitContainerRuntime="1.03498034s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:49:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:49:52.029109634 +0000 UTC m=+276.840457007" watchObservedRunningTime="2026-07-27 21:49:52.030863943 +0000 UTC m=+276.842211316"
Jul 27 21:50:02.999448 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:02.999417    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7"
Jul 27 21:50:05.719415 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:05.719372    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5kvmh"]
Jul 27 21:50:05.730424 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:05.730396    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-5kvmh"
Jul 27 21:50:05.734832 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:05.734804    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 27 21:50:05.736279 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:05.736253    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-kpv8s\""
Jul 27 21:50:05.739194 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:05.738894    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5kvmh"]
Jul 27 21:50:05.774061 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:05.773992    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl9td\" (UniqueName: \"kubernetes.io/projected/bacacc03-d38a-4c8a-9194-d636e52b5215-kube-api-access-rl9td\") pod \"odh-model-controller-7998c67dc4-5kvmh\" (UID: \"bacacc03-d38a-4c8a-9194-d636e52b5215\") " pod="opendatahub/odh-model-controller-7998c67dc4-5kvmh"
Jul 27 21:50:05.774266 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:05.774122    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bacacc03-d38a-4c8a-9194-d636e52b5215-cert\") pod \"odh-model-controller-7998c67dc4-5kvmh\" (UID: \"bacacc03-d38a-4c8a-9194-d636e52b5215\") " pod="opendatahub/odh-model-controller-7998c67dc4-5kvmh"
Jul 27 21:50:05.875275 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:05.875206    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl9td\" (UniqueName: \"kubernetes.io/projected/bacacc03-d38a-4c8a-9194-d636e52b5215-kube-api-access-rl9td\") pod \"odh-model-controller-7998c67dc4-5kvmh\" (UID: \"bacacc03-d38a-4c8a-9194-d636e52b5215\") " pod="opendatahub/odh-model-controller-7998c67dc4-5kvmh"
Jul 27 21:50:05.875476 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:05.875298    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bacacc03-d38a-4c8a-9194-d636e52b5215-cert\") pod \"odh-model-controller-7998c67dc4-5kvmh\" (UID: \"bacacc03-d38a-4c8a-9194-d636e52b5215\") " pod="opendatahub/odh-model-controller-7998c67dc4-5kvmh"
Jul 27 21:50:05.875476 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:50:05.875400    2549 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 27 21:50:05.875476 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:50:05.875467    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bacacc03-d38a-4c8a-9194-d636e52b5215-cert podName:bacacc03-d38a-4c8a-9194-d636e52b5215 nodeName:}" failed. No retries permitted until 2026-07-27 21:50:06.375446142 +0000 UTC m=+291.186793508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bacacc03-d38a-4c8a-9194-d636e52b5215-cert") pod "odh-model-controller-7998c67dc4-5kvmh" (UID: "bacacc03-d38a-4c8a-9194-d636e52b5215") : secret "odh-model-controller-webhook-cert" not found
Jul 27 21:50:05.890852 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:05.890815    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl9td\" (UniqueName: \"kubernetes.io/projected/bacacc03-d38a-4c8a-9194-d636e52b5215-kube-api-access-rl9td\") pod \"odh-model-controller-7998c67dc4-5kvmh\" (UID: \"bacacc03-d38a-4c8a-9194-d636e52b5215\") " pod="opendatahub/odh-model-controller-7998c67dc4-5kvmh"
Jul 27 21:50:06.376023 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.375967    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6n7dt"]
Jul 27 21:50:06.378311 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.378280    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6n7dt"
Jul 27 21:50:06.378872 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.378843    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bacacc03-d38a-4c8a-9194-d636e52b5215-cert\") pod \"odh-model-controller-7998c67dc4-5kvmh\" (UID: \"bacacc03-d38a-4c8a-9194-d636e52b5215\") " pod="opendatahub/odh-model-controller-7998c67dc4-5kvmh"
Jul 27 21:50:06.381349 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.381328    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bacacc03-d38a-4c8a-9194-d636e52b5215-cert\") pod \"odh-model-controller-7998c67dc4-5kvmh\" (UID: \"bacacc03-d38a-4c8a-9194-d636e52b5215\") " pod="opendatahub/odh-model-controller-7998c67dc4-5kvmh"
Jul 27 21:50:06.382260 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.382242    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 27 21:50:06.393146 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.393123    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6n7dt"]
Jul 27 21:50:06.479529 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.479490    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq7h2\" (UniqueName: \"kubernetes.io/projected/15fcbdb9-b762-47ea-bf94-f7d3901f16cd-kube-api-access-bq7h2\") pod \"model-serving-api-54d89bd5fc-6n7dt\" (UID: \"15fcbdb9-b762-47ea-bf94-f7d3901f16cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-6n7dt"
Jul 27 21:50:06.479529 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.479534    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15fcbdb9-b762-47ea-bf94-f7d3901f16cd-tls-certs\") pod \"model-serving-api-54d89bd5fc-6n7dt\" (UID: \"15fcbdb9-b762-47ea-bf94-f7d3901f16cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-6n7dt"
Jul 27 21:50:06.580745 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.580708    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq7h2\" (UniqueName: \"kubernetes.io/projected/15fcbdb9-b762-47ea-bf94-f7d3901f16cd-kube-api-access-bq7h2\") pod \"model-serving-api-54d89bd5fc-6n7dt\" (UID: \"15fcbdb9-b762-47ea-bf94-f7d3901f16cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-6n7dt"
Jul 27 21:50:06.580931 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.580756    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15fcbdb9-b762-47ea-bf94-f7d3901f16cd-tls-certs\") pod \"model-serving-api-54d89bd5fc-6n7dt\" (UID: \"15fcbdb9-b762-47ea-bf94-f7d3901f16cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-6n7dt"
Jul 27 21:50:06.583530 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.583504    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15fcbdb9-b762-47ea-bf94-f7d3901f16cd-tls-certs\") pod \"model-serving-api-54d89bd5fc-6n7dt\" (UID: \"15fcbdb9-b762-47ea-bf94-f7d3901f16cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-6n7dt"
Jul 27 21:50:06.589427 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.589400    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq7h2\" (UniqueName: \"kubernetes.io/projected/15fcbdb9-b762-47ea-bf94-f7d3901f16cd-kube-api-access-bq7h2\") pod \"model-serving-api-54d89bd5fc-6n7dt\" (UID: \"15fcbdb9-b762-47ea-bf94-f7d3901f16cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-6n7dt"
Jul 27 21:50:06.644270 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.644167    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-5kvmh"
Jul 27 21:50:06.697726 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.697691    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6n7dt"
Jul 27 21:50:06.794693 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.794658    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5kvmh"]
Jul 27 21:50:06.798832 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:50:06.798805    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacacc03_d38a_4c8a_9194_d636e52b5215.slice/crio-19004772c7ab8047c6d6e64a0a3720efa1d034165c61df65405627d7e943b25f WatchSource:0}: Error finding container 19004772c7ab8047c6d6e64a0a3720efa1d034165c61df65405627d7e943b25f: Status 404 returned error can't find the container with id 19004772c7ab8047c6d6e64a0a3720efa1d034165c61df65405627d7e943b25f
Jul 27 21:50:06.863662 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:06.863634    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6n7dt"]
Jul 27 21:50:06.866077 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:50:06.866042    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fcbdb9_b762_47ea_bf94_f7d3901f16cd.slice/crio-96134cdf16566c37d81bac038193c0228f836cf1592b378bea6eece346e9386f WatchSource:0}: Error finding container 96134cdf16566c37d81bac038193c0228f836cf1592b378bea6eece346e9386f: Status 404 returned error can't find the container with id 96134cdf16566c37d81bac038193c0228f836cf1592b378bea6eece346e9386f
Jul 27 21:50:07.061351 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:07.061252    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6n7dt" event={"ID":"15fcbdb9-b762-47ea-bf94-f7d3901f16cd","Type":"ContainerStarted","Data":"96134cdf16566c37d81bac038193c0228f836cf1592b378bea6eece346e9386f"}
Jul 27 21:50:07.062301 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:07.062280    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-5kvmh" event={"ID":"bacacc03-d38a-4c8a-9194-d636e52b5215","Type":"ContainerStarted","Data":"19004772c7ab8047c6d6e64a0a3720efa1d034165c61df65405627d7e943b25f"}
Jul 27 21:50:10.078638 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:10.078538    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6n7dt" event={"ID":"15fcbdb9-b762-47ea-bf94-f7d3901f16cd","Type":"ContainerStarted","Data":"9a6288a7e9e5b8d64d73db5efc4e3e3b14fd73ee06817f5e625328791c7d6116"}
Jul 27 21:50:10.079100 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:10.078640    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-6n7dt"
Jul 27 21:50:10.080120 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:10.080094    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-5kvmh" event={"ID":"bacacc03-d38a-4c8a-9194-d636e52b5215","Type":"ContainerStarted","Data":"65ac0567960d70e51a2333a5566525b336c0f34a627c2eabc9a5d74e9f99e780"}
Jul 27 21:50:10.080278 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:10.080258    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-5kvmh"
Jul 27 21:50:10.097159 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:10.097095    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-6n7dt" podStartSLOduration=1.277679592 podStartE2EDuration="4.097072196s" totalImagesPullingTime="2.819392604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:50:10.096061767 +0000 UTC m=+294.907409151" watchObservedRunningTime="2026-07-27 21:50:10.097072196 +0000 UTC m=+294.908419569"
Jul 27 21:50:10.115850 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:10.115794    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-5kvmh" podStartSLOduration=2.179576242 podStartE2EDuration="5.115776821s" totalImagesPullingTime="2.936200579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:50:10.114634297 +0000 UTC m=+294.925981671" watchObservedRunningTime="2026-07-27 21:50:10.115776821 +0000 UTC m=+294.927124196"
Jul 27 21:50:15.561329 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:15.561298    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 21:50:15.566395 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:15.566368    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 21:50:15.573352 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:15.573323    2549 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 21:50:16.046544 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.046511    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"]
Jul 27 21:50:16.049717 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.049696    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"
Jul 27 21:50:16.057713 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.057436    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 21:50:16.057792 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.057740    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 21:50:16.058002 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.057985    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 21:50:16.059801 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.059783    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 21:50:16.062931 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.062905    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"]
Jul 27 21:50:16.121762 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.121728    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dx46t"]
Jul 27 21:50:16.124934 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.124912    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dx46t"
Jul 27 21:50:16.127992 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.127975    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 27 21:50:16.128344 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.128330    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-h8jxn\""
Jul 27 21:50:16.135887 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.135863    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dx46t"]
Jul 27 21:50:16.141965 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.141947    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv"]
Jul 27 21:50:16.145048 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.145029    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv"
Jul 27 21:50:16.147847 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.147830    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2tkf9\""
Jul 27 21:50:16.148208 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.148190    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 27 21:50:16.159657 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.159629    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv"]
Jul 27 21:50:16.171019 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.170992    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-tmp\") pod \"kube-auth-proxy-5794464b4b-4jqss\" (UID: \"b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"
Jul 27 21:50:16.171138 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.171084    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vz65\" (UniqueName: \"kubernetes.io/projected/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-kube-api-access-8vz65\") pod \"kube-auth-proxy-5794464b4b-4jqss\" (UID: \"b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"
Jul 27 21:50:16.171198 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.171183    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-tls-certs\") pod \"kube-auth-proxy-5794464b4b-4jqss\" (UID: \"b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"
Jul 27 21:50:16.272357 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.272315    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw5g5\" (UniqueName: \"kubernetes.io/projected/d09d1f65-5192-4d4e-84de-20e715df224a-kube-api-access-kw5g5\") pod \"kserve-controller-manager-d6f4b65b8-dx46t\" (UID: \"d09d1f65-5192-4d4e-84de-20e715df224a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dx46t"
Jul 27 21:50:16.272553 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.272371    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-tls-certs\") pod \"kube-auth-proxy-5794464b4b-4jqss\" (UID: \"b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"
Jul 27 21:50:16.272553 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.272443    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-tmp\") pod \"kube-auth-proxy-5794464b4b-4jqss\" (UID: \"b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"
Jul 27 21:50:16.272553 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:50:16.272468    2549 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 27 21:50:16.272553 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.272485    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b701741e-22b5-4e43-ae3e-c90642f4a309-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p8tmv\" (UID: \"b701741e-22b5-4e43-ae3e-c90642f4a309\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv"
Jul 27 21:50:16.272553 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.272508    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzlzl\" (UniqueName: \"kubernetes.io/projected/b701741e-22b5-4e43-ae3e-c90642f4a309-kube-api-access-jzlzl\") pod \"llmisvc-controller-manager-777c6bd7d5-p8tmv\" (UID: \"b701741e-22b5-4e43-ae3e-c90642f4a309\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv"
Jul 27 21:50:16.272553 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:50:16.272532    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-tls-certs podName:b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1 nodeName:}" failed. No retries permitted until 2026-07-27 21:50:16.772510391 +0000 UTC m=+301.583857742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-tls-certs") pod "kube-auth-proxy-5794464b4b-4jqss" (UID: "b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1") : secret "kube-auth-proxy-tls" not found
Jul 27 21:50:16.272860 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.272636    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d09d1f65-5192-4d4e-84de-20e715df224a-cert\") pod \"kserve-controller-manager-d6f4b65b8-dx46t\" (UID: \"d09d1f65-5192-4d4e-84de-20e715df224a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dx46t"
Jul 27 21:50:16.272860 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.272669    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vz65\" (UniqueName: \"kubernetes.io/projected/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-kube-api-access-8vz65\") pod \"kube-auth-proxy-5794464b4b-4jqss\" (UID: \"b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"
Jul 27 21:50:16.274905 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.274884    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-tmp\") pod \"kube-auth-proxy-5794464b4b-4jqss\" (UID: \"b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"
Jul 27 21:50:16.291331 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.291299    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vz65\" (UniqueName: \"kubernetes.io/projected/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-kube-api-access-8vz65\") pod \"kube-auth-proxy-5794464b4b-4jqss\" (UID: \"b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"
Jul 27 21:50:16.373273 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.373206    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw5g5\" (UniqueName: \"kubernetes.io/projected/d09d1f65-5192-4d4e-84de-20e715df224a-kube-api-access-kw5g5\") pod \"kserve-controller-manager-d6f4b65b8-dx46t\" (UID: \"d09d1f65-5192-4d4e-84de-20e715df224a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dx46t"
Jul 27 21:50:16.373461 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.373312    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b701741e-22b5-4e43-ae3e-c90642f4a309-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p8tmv\" (UID: \"b701741e-22b5-4e43-ae3e-c90642f4a309\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv"
Jul 27 21:50:16.373461 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.373335    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlzl\" (UniqueName: \"kubernetes.io/projected/b701741e-22b5-4e43-ae3e-c90642f4a309-kube-api-access-jzlzl\") pod \"llmisvc-controller-manager-777c6bd7d5-p8tmv\" (UID: \"b701741e-22b5-4e43-ae3e-c90642f4a309\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv"
Jul 27 21:50:16.373461 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.373357    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d09d1f65-5192-4d4e-84de-20e715df224a-cert\") pod \"kserve-controller-manager-d6f4b65b8-dx46t\" (UID: \"d09d1f65-5192-4d4e-84de-20e715df224a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dx46t"
Jul 27 21:50:16.373579 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:50:16.373461    2549 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 27 21:50:16.373579 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:50:16.373531    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d09d1f65-5192-4d4e-84de-20e715df224a-cert podName:d09d1f65-5192-4d4e-84de-20e715df224a nodeName:}" failed. No retries permitted until 2026-07-27 21:50:16.873514072 +0000 UTC m=+301.684861436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d09d1f65-5192-4d4e-84de-20e715df224a-cert") pod "kserve-controller-manager-d6f4b65b8-dx46t" (UID: "d09d1f65-5192-4d4e-84de-20e715df224a") : secret "kserve-webhook-server-cert" not found
Jul 27 21:50:16.375877 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.375854    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b701741e-22b5-4e43-ae3e-c90642f4a309-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p8tmv\" (UID: \"b701741e-22b5-4e43-ae3e-c90642f4a309\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv"
Jul 27 21:50:16.386481 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.386449    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlzl\" (UniqueName: \"kubernetes.io/projected/b701741e-22b5-4e43-ae3e-c90642f4a309-kube-api-access-jzlzl\") pod \"llmisvc-controller-manager-777c6bd7d5-p8tmv\" (UID: \"b701741e-22b5-4e43-ae3e-c90642f4a309\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv"
Jul 27 21:50:16.387569 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.387547    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw5g5\" (UniqueName: \"kubernetes.io/projected/d09d1f65-5192-4d4e-84de-20e715df224a-kube-api-access-kw5g5\") pod \"kserve-controller-manager-d6f4b65b8-dx46t\" (UID: \"d09d1f65-5192-4d4e-84de-20e715df224a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dx46t"
Jul 27 21:50:16.454603 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.454571    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv"
Jul 27 21:50:16.592795 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.592770    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv"]
Jul 27 21:50:16.595123 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:50:16.595095    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb701741e_22b5_4e43_ae3e_c90642f4a309.slice/crio-bf977df5c265ecbd33f0a2723bd5bf94dd165db283f99656d9328b273acd7641 WatchSource:0}: Error finding container bf977df5c265ecbd33f0a2723bd5bf94dd165db283f99656d9328b273acd7641: Status 404 returned error can't find the container with id bf977df5c265ecbd33f0a2723bd5bf94dd165db283f99656d9328b273acd7641
Jul 27 21:50:16.777341 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.777218    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-tls-certs\") pod \"kube-auth-proxy-5794464b4b-4jqss\" (UID: \"b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"
Jul 27 21:50:16.777470 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:50:16.777355    2549 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 27 21:50:16.777470 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:50:16.777417    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-tls-certs podName:b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1 nodeName:}" failed. No retries permitted until 2026-07-27 21:50:17.777402973 +0000 UTC m=+302.588750322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-tls-certs") pod "kube-auth-proxy-5794464b4b-4jqss" (UID: "b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1") : secret "kube-auth-proxy-tls" not found
Jul 27 21:50:16.878458 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.878422    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d09d1f65-5192-4d4e-84de-20e715df224a-cert\") pod \"kserve-controller-manager-d6f4b65b8-dx46t\" (UID: \"d09d1f65-5192-4d4e-84de-20e715df224a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dx46t"
Jul 27 21:50:16.880885 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:16.880866    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d09d1f65-5192-4d4e-84de-20e715df224a-cert\") pod \"kserve-controller-manager-d6f4b65b8-dx46t\" (UID: \"d09d1f65-5192-4d4e-84de-20e715df224a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dx46t"
Jul 27 21:50:17.035189 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:17.035107    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dx46t"
Jul 27 21:50:17.108450 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:17.108415    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv" event={"ID":"b701741e-22b5-4e43-ae3e-c90642f4a309","Type":"ContainerStarted","Data":"bf977df5c265ecbd33f0a2723bd5bf94dd165db283f99656d9328b273acd7641"}
Jul 27 21:50:17.185788 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:17.185756    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dx46t"]
Jul 27 21:50:17.188801 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:50:17.188772    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09d1f65_5192_4d4e_84de_20e715df224a.slice/crio-d1cdf0bd181695e96550fbf648783963c713617108975f05f17f7a52c5dbd809 WatchSource:0}: Error finding container d1cdf0bd181695e96550fbf648783963c713617108975f05f17f7a52c5dbd809: Status 404 returned error can't find the container with id d1cdf0bd181695e96550fbf648783963c713617108975f05f17f7a52c5dbd809
Jul 27 21:50:17.191897 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:17.191273    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:50:17.786502 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:17.786260    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-tls-certs\") pod \"kube-auth-proxy-5794464b4b-4jqss\" (UID: \"b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"
Jul 27 21:50:17.789519 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:17.789491    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1-tls-certs\") pod \"kube-auth-proxy-5794464b4b-4jqss\" (UID: \"b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"
Jul 27 21:50:17.860468 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:17.860434    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"
Jul 27 21:50:18.116512 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.116401    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dx46t" event={"ID":"d09d1f65-5192-4d4e-84de-20e715df224a","Type":"ContainerStarted","Data":"d1cdf0bd181695e96550fbf648783963c713617108975f05f17f7a52c5dbd809"}
Jul 27 21:50:18.253782 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.253745    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5794464b4b-4jqss"]
Jul 27 21:50:18.257865 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:50:18.257833    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fe3c49_3ec7_4362_9a66_ec3f2d1fa6f1.slice/crio-7603d469d9c4056930f0cdda48350de3ecde8756944f23e712f02ba2c8127194 WatchSource:0}: Error finding container 7603d469d9c4056930f0cdda48350de3ecde8756944f23e712f02ba2c8127194: Status 404 returned error can't find the container with id 7603d469d9c4056930f0cdda48350de3ecde8756944f23e712f02ba2c8127194
Jul 27 21:50:18.396202 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.396166    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"]
Jul 27 21:50:18.400044 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.400019    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.406947 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.406923    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 27 21:50:18.417101 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.417067    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c5v8d\""
Jul 27 21:50:18.428006 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.427971    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 27 21:50:18.428185 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.428144    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 21:50:18.428276 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.428144    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 27 21:50:18.436112 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.436078    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"]
Jul 27 21:50:18.493109 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.493066    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35d1ae81-f38e-4696-b7d6-60045fea8c7b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.493109 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.493108    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/35d1ae81-f38e-4696-b7d6-60045fea8c7b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.493374 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.493137    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/35d1ae81-f38e-4696-b7d6-60045fea8c7b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.493374 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.493163    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d1ae81-f38e-4696-b7d6-60045fea8c7b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.493374 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.493272    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/35d1ae81-f38e-4696-b7d6-60045fea8c7b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.493374 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.493329    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlfnf\" (UniqueName: \"kubernetes.io/projected/35d1ae81-f38e-4696-b7d6-60045fea8c7b-kube-api-access-wlfnf\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.493545 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.493384    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d1ae81-f38e-4696-b7d6-60045fea8c7b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.594239 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.594183    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d1ae81-f38e-4696-b7d6-60045fea8c7b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.594428 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.594291    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35d1ae81-f38e-4696-b7d6-60045fea8c7b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.594428 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.594323    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/35d1ae81-f38e-4696-b7d6-60045fea8c7b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.594428 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.594350    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/35d1ae81-f38e-4696-b7d6-60045fea8c7b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.594428 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.594379    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d1ae81-f38e-4696-b7d6-60045fea8c7b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.594428 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.594418    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/35d1ae81-f38e-4696-b7d6-60045fea8c7b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.594691 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.594452    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlfnf\" (UniqueName: \"kubernetes.io/projected/35d1ae81-f38e-4696-b7d6-60045fea8c7b-kube-api-access-wlfnf\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.595793 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.595761    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/35d1ae81-f38e-4696-b7d6-60045fea8c7b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.597392 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.597351    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/35d1ae81-f38e-4696-b7d6-60045fea8c7b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.597500 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.597416    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d1ae81-f38e-4696-b7d6-60045fea8c7b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.597567 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.597498    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/35d1ae81-f38e-4696-b7d6-60045fea8c7b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.597627 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.597565    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35d1ae81-f38e-4696-b7d6-60045fea8c7b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.609052 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.609019    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlfnf\" (UniqueName: \"kubernetes.io/projected/35d1ae81-f38e-4696-b7d6-60045fea8c7b-kube-api-access-wlfnf\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.609910 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.609886    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d1ae81-f38e-4696-b7d6-60045fea8c7b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-gwhvs\" (UID: \"35d1ae81-f38e-4696-b7d6-60045fea8c7b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.720123 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.720032    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:18.901587 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:50:18.901548    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d1ae81_f38e_4696_b7d6_60045fea8c7b.slice/crio-b83581a8b27451e8311b4886189e1cc39f12724100b7711b9a1e259f3301c155 WatchSource:0}: Error finding container b83581a8b27451e8311b4886189e1cc39f12724100b7711b9a1e259f3301c155: Status 404 returned error can't find the container with id b83581a8b27451e8311b4886189e1cc39f12724100b7711b9a1e259f3301c155
Jul 27 21:50:18.901998 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:18.901919    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"]
Jul 27 21:50:19.125682 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:19.125637    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs" event={"ID":"35d1ae81-f38e-4696-b7d6-60045fea8c7b","Type":"ContainerStarted","Data":"b83581a8b27451e8311b4886189e1cc39f12724100b7711b9a1e259f3301c155"}
Jul 27 21:50:19.127416 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:19.127366    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss" event={"ID":"b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1","Type":"ContainerStarted","Data":"7603d469d9c4056930f0cdda48350de3ecde8756944f23e712f02ba2c8127194"}
Jul 27 21:50:21.087253 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:21.087187    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-5kvmh"
Jul 27 21:50:21.092925 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:21.092372    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-6n7dt"
Jul 27 21:50:22.160378 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:22.160342    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv" event={"ID":"b701741e-22b5-4e43-ae3e-c90642f4a309","Type":"ContainerStarted","Data":"02223b0b9df80e9f63ced406c11f6895161763250628ee5132fbd6b1e874bf21"}
Jul 27 21:50:22.160829 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:22.160442    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv"
Jul 27 21:50:22.162020 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:22.161992    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss" event={"ID":"b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1","Type":"ContainerStarted","Data":"a33e2f7ddb6b2d5ec91298c1ea6da0258723073ab545d4dd7289b2dcc174ba26"}
Jul 27 21:50:22.163553 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:22.163530    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dx46t" event={"ID":"d09d1f65-5192-4d4e-84de-20e715df224a","Type":"ContainerStarted","Data":"9c406f0aaa2fd9c539b7aed16d14d13bbf22c800b1a23712b04025e247211c61"}
Jul 27 21:50:22.163648 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:22.163624    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dx46t"
Jul 27 21:50:22.179252 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:22.179187    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv" podStartSLOduration=1.747541061 podStartE2EDuration="6.179171027s" totalImagesPullingTime="4.431629966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:50:22.178217426 +0000 UTC m=+306.989564799" watchObservedRunningTime="2026-07-27 21:50:22.179171027 +0000 UTC m=+306.990518400"
Jul 27 21:50:22.195540 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:22.195347    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5794464b4b-4jqss" podStartSLOduration=3.419456016 podStartE2EDuration="6.195328991s" totalImagesPullingTime="2.775872975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:50:22.194187378 +0000 UTC m=+307.005534752" watchObservedRunningTime="2026-07-27 21:50:22.195328991 +0000 UTC m=+307.006676367"
Jul 27 21:50:22.214591 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:22.214533    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dx46t" podStartSLOduration=2.376312111 podStartE2EDuration="6.21451282s" totalImagesPullingTime="3.838200709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:50:22.214074687 +0000 UTC m=+307.025422062" watchObservedRunningTime="2026-07-27 21:50:22.21451282 +0000 UTC m=+307.025860194"
Jul 27 21:50:23.027063 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:23.027002    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:50:23.027165 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:23.027093    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:50:23.169626 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:23.169584    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs" event={"ID":"35d1ae81-f38e-4696-b7d6-60045fea8c7b","Type":"ContainerStarted","Data":"078ca09b7ab240ef552ae6d248ec365867c260ad31ee0283924699be20366457"}
Jul 27 21:50:23.170046 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:23.169747    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:23.195947 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:23.195889    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs" podStartSLOduration=1.073083193 podStartE2EDuration="5.195872961s" totalImagesPullingTime="4.122789768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:50:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:50:23.193175485 +0000 UTC m=+308.004522911" watchObservedRunningTime="2026-07-27 21:50:23.195872961 +0000 UTC m=+308.007220333"
Jul 27 21:50:24.175824 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:24.175796    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gwhvs"
Jul 27 21:50:53.172392 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:53.172359    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p8tmv"
Jul 27 21:50:53.175541 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:50:53.175518    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dx46t"
Jul 27 21:51:45.579992 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:45.579955    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qqsp"]
Jul 27 21:51:45.582118 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:45.582102    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qqsp"
Jul 27 21:51:45.585655 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:45.585547    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 27 21:51:45.586704 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:45.586669    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 21:51:45.590783 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:45.587138    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9b5xl\""
Jul 27 21:51:45.590783 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:45.587434    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 21:51:45.604668 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:45.604630    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66bh6\" (UniqueName: \"kubernetes.io/projected/e36fa849-8364-4983-a9cf-5ff5fca2e211-kube-api-access-66bh6\") pod \"dns-operator-controller-manager-864dd78c4c-5qqsp\" (UID: \"e36fa849-8364-4983-a9cf-5ff5fca2e211\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qqsp"
Jul 27 21:51:45.606749 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:45.606723    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qqsp"]
Jul 27 21:51:45.705298 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:45.705263    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66bh6\" (UniqueName: \"kubernetes.io/projected/e36fa849-8364-4983-a9cf-5ff5fca2e211-kube-api-access-66bh6\") pod \"dns-operator-controller-manager-864dd78c4c-5qqsp\" (UID: \"e36fa849-8364-4983-a9cf-5ff5fca2e211\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qqsp"
Jul 27 21:51:45.718775 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:45.718747    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66bh6\" (UniqueName: \"kubernetes.io/projected/e36fa849-8364-4983-a9cf-5ff5fca2e211-kube-api-access-66bh6\") pod \"dns-operator-controller-manager-864dd78c4c-5qqsp\" (UID: \"e36fa849-8364-4983-a9cf-5ff5fca2e211\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qqsp"
Jul 27 21:51:45.898076 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:45.898034    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qqsp"
Jul 27 21:51:46.040371 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:46.040339    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qqsp"]
Jul 27 21:51:46.043587 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:51:46.043562    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode36fa849_8364_4983_a9cf_5ff5fca2e211.slice/crio-9e995716a82323ee1159e32d858705e010fe5c5c0d604bb1c3839624f6d76829 WatchSource:0}: Error finding container 9e995716a82323ee1159e32d858705e010fe5c5c0d604bb1c3839624f6d76829: Status 404 returned error can't find the container with id 9e995716a82323ee1159e32d858705e010fe5c5c0d604bb1c3839624f6d76829
Jul 27 21:51:46.530447 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:46.530410    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qqsp" event={"ID":"e36fa849-8364-4983-a9cf-5ff5fca2e211","Type":"ContainerStarted","Data":"9e995716a82323ee1159e32d858705e010fe5c5c0d604bb1c3839624f6d76829"}
Jul 27 21:51:48.540919 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:48.540825    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qqsp" event={"ID":"e36fa849-8364-4983-a9cf-5ff5fca2e211","Type":"ContainerStarted","Data":"dab7c7975939185f299eaceda3a09a82d0f3916fec1db029439001010848461d"}
Jul 27 21:51:48.541300 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:48.540920    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qqsp"
Jul 27 21:51:48.562969 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:48.562916    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qqsp" podStartSLOduration=1.475994956 podStartE2EDuration="3.56289763s" totalImagesPullingTime="2.086902674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:51:48.56143408 +0000 UTC m=+393.372781445" watchObservedRunningTime="2026-07-27 21:51:48.56289763 +0000 UTC m=+393.374245001"
Jul 27 21:51:59.548313 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:51:59.548280    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qqsp"
Jul 27 21:52:08.987298 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:08.987261    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"]
Jul 27 21:52:08.990639 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:08.990621    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"
Jul 27 21:52:08.993864 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:08.993833    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lv7rp\""
Jul 27 21:52:09.004003 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.003974    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"]
Jul 27 21:52:09.097030 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.096984    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvb4z\" (UniqueName: \"kubernetes.io/projected/541290d0-b6f9-4ffd-8560-4599614e466c-kube-api-access-kvb4z\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-696h2\" (UID: \"541290d0-b6f9-4ffd-8560-4599614e466c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"
Jul 27 21:52:09.097030 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.097036    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/541290d0-b6f9-4ffd-8560-4599614e466c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-696h2\" (UID: \"541290d0-b6f9-4ffd-8560-4599614e466c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"
Jul 27 21:52:09.198143 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.198108    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvb4z\" (UniqueName: \"kubernetes.io/projected/541290d0-b6f9-4ffd-8560-4599614e466c-kube-api-access-kvb4z\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-696h2\" (UID: \"541290d0-b6f9-4ffd-8560-4599614e466c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"
Jul 27 21:52:09.198143 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.198146    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/541290d0-b6f9-4ffd-8560-4599614e466c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-696h2\" (UID: \"541290d0-b6f9-4ffd-8560-4599614e466c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"
Jul 27 21:52:09.198508 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.198490    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/541290d0-b6f9-4ffd-8560-4599614e466c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-696h2\" (UID: \"541290d0-b6f9-4ffd-8560-4599614e466c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"
Jul 27 21:52:09.208177 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.208145    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvb4z\" (UniqueName: \"kubernetes.io/projected/541290d0-b6f9-4ffd-8560-4599614e466c-kube-api-access-kvb4z\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-696h2\" (UID: \"541290d0-b6f9-4ffd-8560-4599614e466c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"
Jul 27 21:52:09.300646 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.300565    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"
Jul 27 21:52:09.443893 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.443870    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"]
Jul 27 21:52:09.446176 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:52:09.446152    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541290d0_b6f9_4ffd_8560_4599614e466c.slice/crio-feeaf05580612740f593115e0a9d51ab83536464abee9e81900efa27a463f921 WatchSource:0}: Error finding container feeaf05580612740f593115e0a9d51ab83536464abee9e81900efa27a463f921: Status 404 returned error can't find the container with id feeaf05580612740f593115e0a9d51ab83536464abee9e81900efa27a463f921
Jul 27 21:52:09.638051 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.638019    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2" event={"ID":"541290d0-b6f9-4ffd-8560-4599614e466c","Type":"ContainerStarted","Data":"feeaf05580612740f593115e0a9d51ab83536464abee9e81900efa27a463f921"}
Jul 27 21:52:09.659497 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.659464    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"]
Jul 27 21:52:09.664456 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.664428    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"]
Jul 27 21:52:09.693447 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.693411    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"]
Jul 27 21:52:09.697858 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.697828    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"
Jul 27 21:52:09.716175 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.716146    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"]
Jul 27 21:52:09.802972 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.802927    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8m7g\" (UniqueName: \"kubernetes.io/projected/70454e27-2e97-447e-ab27-454b990273f4-kube-api-access-z8m7g\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7gzwx\" (UID: \"70454e27-2e97-447e-ab27-454b990273f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"
Jul 27 21:52:09.803186 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.803066    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70454e27-2e97-447e-ab27-454b990273f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7gzwx\" (UID: \"70454e27-2e97-447e-ab27-454b990273f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"
Jul 27 21:52:09.904703 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.904623    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8m7g\" (UniqueName: \"kubernetes.io/projected/70454e27-2e97-447e-ab27-454b990273f4-kube-api-access-z8m7g\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7gzwx\" (UID: \"70454e27-2e97-447e-ab27-454b990273f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"
Jul 27 21:52:09.904703 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.904663    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70454e27-2e97-447e-ab27-454b990273f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7gzwx\" (UID: \"70454e27-2e97-447e-ab27-454b990273f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"
Jul 27 21:52:09.905010 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.904992    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70454e27-2e97-447e-ab27-454b990273f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7gzwx\" (UID: \"70454e27-2e97-447e-ab27-454b990273f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"
Jul 27 21:52:09.915202 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:09.915175    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8m7g\" (UniqueName: \"kubernetes.io/projected/70454e27-2e97-447e-ab27-454b990273f4-kube-api-access-z8m7g\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7gzwx\" (UID: \"70454e27-2e97-447e-ab27-454b990273f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"
Jul 27 21:52:10.010776 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:10.010738    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"
Jul 27 21:52:10.237581 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:10.237461    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"]
Jul 27 21:52:10.311299 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:52:10.311264    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70454e27_2e97_447e_ab27_454b990273f4.slice/crio-b44b40c52b39acd1b14043cf22a3305187625339b11c65e28a251a2c310d20d4 WatchSource:0}: Error finding container b44b40c52b39acd1b14043cf22a3305187625339b11c65e28a251a2c310d20d4: Status 404 returned error can't find the container with id b44b40c52b39acd1b14043cf22a3305187625339b11c65e28a251a2c310d20d4
Jul 27 21:52:10.645701 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:10.645661    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx" event={"ID":"70454e27-2e97-447e-ab27-454b990273f4","Type":"ContainerStarted","Data":"b44b40c52b39acd1b14043cf22a3305187625339b11c65e28a251a2c310d20d4"}
Jul 27 21:52:14.668132 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:14.668091    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2" podUID="541290d0-b6f9-4ffd-8560-4599614e466c" containerName="manager" containerID="cri-o://33e6a0a755a67666e4b40597a6f21be5d403d30749fd6b533a3ff35c3254e622" gracePeriod=2
Jul 27 21:52:14.670054 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:14.670016    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx" event={"ID":"70454e27-2e97-447e-ab27-454b990273f4","Type":"ContainerStarted","Data":"918477924f1f7ed06227fa6f0f97502e9a07d05159bc3c457218fcc88e93bd79"}
Jul 27 21:52:14.670165 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:14.670101    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"
Jul 27 21:52:14.693356 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:14.693309    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx" podStartSLOduration=2.374745125 podStartE2EDuration="5.693295063s" totalImagesPullingTime="3.318549938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:52:14.692868605 +0000 UTC m=+419.504215978" watchObservedRunningTime="2026-07-27 21:52:14.693295063 +0000 UTC m=+419.504642434"
Jul 27 21:52:14.913694 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:14.913671    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"
Jul 27 21:52:14.916312 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:52:14.916282    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-696h2\" is forbidden: User \"system:node:ip-10-0-138-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-124.ec2.internal' and this object" podUID="541290d0-b6f9-4ffd-8560-4599614e466c" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"
Jul 27 21:52:15.052945 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:15.052857    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvb4z\" (UniqueName: \"kubernetes.io/projected/541290d0-b6f9-4ffd-8560-4599614e466c-kube-api-access-kvb4z\") pod \"541290d0-b6f9-4ffd-8560-4599614e466c\" (UID: \"541290d0-b6f9-4ffd-8560-4599614e466c\") "
Jul 27 21:52:15.052945 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:15.052927    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/541290d0-b6f9-4ffd-8560-4599614e466c-extensions-socket-volume\") pod \"541290d0-b6f9-4ffd-8560-4599614e466c\" (UID: \"541290d0-b6f9-4ffd-8560-4599614e466c\") "
Jul 27 21:52:15.053214 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:15.053191    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541290d0-b6f9-4ffd-8560-4599614e466c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "541290d0-b6f9-4ffd-8560-4599614e466c" (UID: "541290d0-b6f9-4ffd-8560-4599614e466c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:52:15.055291 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:15.055263    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541290d0-b6f9-4ffd-8560-4599614e466c-kube-api-access-kvb4z" (OuterVolumeSpecName: "kube-api-access-kvb4z") pod "541290d0-b6f9-4ffd-8560-4599614e466c" (UID: "541290d0-b6f9-4ffd-8560-4599614e466c"). InnerVolumeSpecName "kube-api-access-kvb4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:52:15.153906 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:15.153855    2549 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/541290d0-b6f9-4ffd-8560-4599614e466c-extensions-socket-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:52:15.153906 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:15.153897    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvb4z\" (UniqueName: \"kubernetes.io/projected/541290d0-b6f9-4ffd-8560-4599614e466c-kube-api-access-kvb4z\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:52:15.675658 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:15.675624    2549 generic.go:356] "Generic (PLEG): container finished" podID="541290d0-b6f9-4ffd-8560-4599614e466c" containerID="33e6a0a755a67666e4b40597a6f21be5d403d30749fd6b533a3ff35c3254e622" exitCode=2
Jul 27 21:52:15.676088 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:15.675676    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"
Jul 27 21:52:15.676088 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:15.675725    2549 scope.go:117] "RemoveContainer" containerID="33e6a0a755a67666e4b40597a6f21be5d403d30749fd6b533a3ff35c3254e622"
Jul 27 21:52:15.678341 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:52:15.678312    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-696h2\" is forbidden: User \"system:node:ip-10-0-138-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-124.ec2.internal' and this object" podUID="541290d0-b6f9-4ffd-8560-4599614e466c" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"
Jul 27 21:52:15.684805 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:15.684780    2549 scope.go:117] "RemoveContainer" containerID="33e6a0a755a67666e4b40597a6f21be5d403d30749fd6b533a3ff35c3254e622"
Jul 27 21:52:15.685101 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:52:15.685083    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e6a0a755a67666e4b40597a6f21be5d403d30749fd6b533a3ff35c3254e622\": container with ID starting with 33e6a0a755a67666e4b40597a6f21be5d403d30749fd6b533a3ff35c3254e622 not found: ID does not exist" containerID="33e6a0a755a67666e4b40597a6f21be5d403d30749fd6b533a3ff35c3254e622"
Jul 27 21:52:15.685177 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:15.685111    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e6a0a755a67666e4b40597a6f21be5d403d30749fd6b533a3ff35c3254e622"} err="failed to get container status \"33e6a0a755a67666e4b40597a6f21be5d403d30749fd6b533a3ff35c3254e622\": rpc error: code = NotFound desc = could not find container \"33e6a0a755a67666e4b40597a6f21be5d403d30749fd6b533a3ff35c3254e622\": container with ID starting with 33e6a0a755a67666e4b40597a6f21be5d403d30749fd6b533a3ff35c3254e622 not found: ID does not exist"
Jul 27 21:52:15.685773 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:52:15.685751    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-696h2\" is forbidden: User \"system:node:ip-10-0-138-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-124.ec2.internal' and this object" podUID="541290d0-b6f9-4ffd-8560-4599614e466c" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"
Jul 27 21:52:15.699189 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:52:15.699160    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-696h2\" is forbidden: User \"system:node:ip-10-0-138-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-124.ec2.internal' and this object" podUID="541290d0-b6f9-4ffd-8560-4599614e466c" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-696h2"
Jul 27 21:52:15.701426 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:15.701402    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="541290d0-b6f9-4ffd-8560-4599614e466c" path="/var/lib/kubelet/pods/541290d0-b6f9-4ffd-8560-4599614e466c/volumes"
Jul 27 21:52:25.678247 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:25.678192    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"
Jul 27 21:52:38.132617 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.132541    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"]
Jul 27 21:52:38.133049 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.132762    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx" podUID="70454e27-2e97-447e-ab27-454b990273f4" containerName="manager" containerID="cri-o://918477924f1f7ed06227fa6f0f97502e9a07d05159bc3c457218fcc88e93bd79" gracePeriod=10
Jul 27 21:52:38.388034 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.387979    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"
Jul 27 21:52:38.446696 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.446664    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8m7g\" (UniqueName: \"kubernetes.io/projected/70454e27-2e97-447e-ab27-454b990273f4-kube-api-access-z8m7g\") pod \"70454e27-2e97-447e-ab27-454b990273f4\" (UID: \"70454e27-2e97-447e-ab27-454b990273f4\") "
Jul 27 21:52:38.446840 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.446733    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70454e27-2e97-447e-ab27-454b990273f4-extensions-socket-volume\") pod \"70454e27-2e97-447e-ab27-454b990273f4\" (UID: \"70454e27-2e97-447e-ab27-454b990273f4\") "
Jul 27 21:52:38.447089 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.447066    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70454e27-2e97-447e-ab27-454b990273f4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "70454e27-2e97-447e-ab27-454b990273f4" (UID: "70454e27-2e97-447e-ab27-454b990273f4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:52:38.448838 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.448817    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70454e27-2e97-447e-ab27-454b990273f4-kube-api-access-z8m7g" (OuterVolumeSpecName: "kube-api-access-z8m7g") pod "70454e27-2e97-447e-ab27-454b990273f4" (UID: "70454e27-2e97-447e-ab27-454b990273f4"). InnerVolumeSpecName "kube-api-access-z8m7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:52:38.548271 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.548212    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8m7g\" (UniqueName: \"kubernetes.io/projected/70454e27-2e97-447e-ab27-454b990273f4-kube-api-access-z8m7g\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:52:38.548271 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.548276    2549 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70454e27-2e97-447e-ab27-454b990273f4-extensions-socket-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:52:38.778680 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.778589    2549 generic.go:356] "Generic (PLEG): container finished" podID="70454e27-2e97-447e-ab27-454b990273f4" containerID="918477924f1f7ed06227fa6f0f97502e9a07d05159bc3c457218fcc88e93bd79" exitCode=0
Jul 27 21:52:38.778680 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.778640    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"
Jul 27 21:52:38.778903 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.778641    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx" event={"ID":"70454e27-2e97-447e-ab27-454b990273f4","Type":"ContainerDied","Data":"918477924f1f7ed06227fa6f0f97502e9a07d05159bc3c457218fcc88e93bd79"}
Jul 27 21:52:38.778903 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.778768    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx" event={"ID":"70454e27-2e97-447e-ab27-454b990273f4","Type":"ContainerDied","Data":"b44b40c52b39acd1b14043cf22a3305187625339b11c65e28a251a2c310d20d4"}
Jul 27 21:52:38.778903 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.778797    2549 scope.go:117] "RemoveContainer" containerID="918477924f1f7ed06227fa6f0f97502e9a07d05159bc3c457218fcc88e93bd79"
Jul 27 21:52:38.787880 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.787862    2549 scope.go:117] "RemoveContainer" containerID="918477924f1f7ed06227fa6f0f97502e9a07d05159bc3c457218fcc88e93bd79"
Jul 27 21:52:38.788136 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:52:38.788117    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918477924f1f7ed06227fa6f0f97502e9a07d05159bc3c457218fcc88e93bd79\": container with ID starting with 918477924f1f7ed06227fa6f0f97502e9a07d05159bc3c457218fcc88e93bd79 not found: ID does not exist" containerID="918477924f1f7ed06227fa6f0f97502e9a07d05159bc3c457218fcc88e93bd79"
Jul 27 21:52:38.788199 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.788146    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918477924f1f7ed06227fa6f0f97502e9a07d05159bc3c457218fcc88e93bd79"} err="failed to get container status \"918477924f1f7ed06227fa6f0f97502e9a07d05159bc3c457218fcc88e93bd79\": rpc error: code = NotFound desc = could not find container \"918477924f1f7ed06227fa6f0f97502e9a07d05159bc3c457218fcc88e93bd79\": container with ID starting with 918477924f1f7ed06227fa6f0f97502e9a07d05159bc3c457218fcc88e93bd79 not found: ID does not exist"
Jul 27 21:52:38.801648 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.801619    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"]
Jul 27 21:52:38.807474 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:38.807447    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7gzwx"]
Jul 27 21:52:39.700128 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:39.700090    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70454e27-2e97-447e-ab27-454b990273f4" path="/var/lib/kubelet/pods/70454e27-2e97-447e-ab27-454b990273f4/volumes"
Jul 27 21:52:56.041112 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.041078    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"]
Jul 27 21:52:56.041506 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.041423    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="541290d0-b6f9-4ffd-8560-4599614e466c" containerName="manager"
Jul 27 21:52:56.041506 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.041434    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="541290d0-b6f9-4ffd-8560-4599614e466c" containerName="manager"
Jul 27 21:52:56.041506 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.041449    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70454e27-2e97-447e-ab27-454b990273f4" containerName="manager"
Jul 27 21:52:56.041506 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.041455    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="70454e27-2e97-447e-ab27-454b990273f4" containerName="manager"
Jul 27 21:52:56.041628 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.041522    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="541290d0-b6f9-4ffd-8560-4599614e466c" containerName="manager"
Jul 27 21:52:56.041628 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.041530    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="70454e27-2e97-447e-ab27-454b990273f4" containerName="manager"
Jul 27 21:52:56.045574 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.045557    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"
Jul 27 21:52:56.048402 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.048382    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 21:52:56.048508 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.048383    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-r79nw\""
Jul 27 21:52:56.052050 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.052027    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"]
Jul 27 21:52:56.095878 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.095839    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7986e592-cb59-4d56-aca9-d2f1a7fd426a-config-file\") pod \"limitador-limitador-64c94fd64f-5ss9q\" (UID: \"7986e592-cb59-4d56-aca9-d2f1a7fd426a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"
Jul 27 21:52:56.095878 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.095877    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4448\" (UniqueName: \"kubernetes.io/projected/7986e592-cb59-4d56-aca9-d2f1a7fd426a-kube-api-access-d4448\") pod \"limitador-limitador-64c94fd64f-5ss9q\" (UID: \"7986e592-cb59-4d56-aca9-d2f1a7fd426a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"
Jul 27 21:52:56.138818 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.138781    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"]
Jul 27 21:52:56.196558 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.196512    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7986e592-cb59-4d56-aca9-d2f1a7fd426a-config-file\") pod \"limitador-limitador-64c94fd64f-5ss9q\" (UID: \"7986e592-cb59-4d56-aca9-d2f1a7fd426a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"
Jul 27 21:52:56.196558 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.196557    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4448\" (UniqueName: \"kubernetes.io/projected/7986e592-cb59-4d56-aca9-d2f1a7fd426a-kube-api-access-d4448\") pod \"limitador-limitador-64c94fd64f-5ss9q\" (UID: \"7986e592-cb59-4d56-aca9-d2f1a7fd426a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"
Jul 27 21:52:56.197168 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.197143    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7986e592-cb59-4d56-aca9-d2f1a7fd426a-config-file\") pod \"limitador-limitador-64c94fd64f-5ss9q\" (UID: \"7986e592-cb59-4d56-aca9-d2f1a7fd426a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"
Jul 27 21:52:56.205378 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.205353    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4448\" (UniqueName: \"kubernetes.io/projected/7986e592-cb59-4d56-aca9-d2f1a7fd426a-kube-api-access-d4448\") pod \"limitador-limitador-64c94fd64f-5ss9q\" (UID: \"7986e592-cb59-4d56-aca9-d2f1a7fd426a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"
Jul 27 21:52:56.356256 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.356195    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"
Jul 27 21:52:56.506644 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.506617    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"]
Jul 27 21:52:56.508571 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:52:56.508540    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7986e592_cb59_4d56_aca9_d2f1a7fd426a.slice/crio-ee3e92465486273272674dbdd0f245962c8492719a963ff9990df72b8700ad54 WatchSource:0}: Error finding container ee3e92465486273272674dbdd0f245962c8492719a963ff9990df72b8700ad54: Status 404 returned error can't find the container with id ee3e92465486273272674dbdd0f245962c8492719a963ff9990df72b8700ad54
Jul 27 21:52:56.866414 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:56.866376    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q" event={"ID":"7986e592-cb59-4d56-aca9-d2f1a7fd426a","Type":"ContainerStarted","Data":"ee3e92465486273272674dbdd0f245962c8492719a963ff9990df72b8700ad54"}
Jul 27 21:52:59.883385 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:59.883351    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q" event={"ID":"7986e592-cb59-4d56-aca9-d2f1a7fd426a","Type":"ContainerStarted","Data":"8b21c850c3efa4af5bbbfb69124821f97695482064066ed9575a8620d7c8a065"}
Jul 27 21:52:59.883826 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:59.883482    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"
Jul 27 21:52:59.902026 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:52:59.901974    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q" podStartSLOduration=1.2730241150000001 podStartE2EDuration="3.901958966s" totalImagesPullingTime="2.628934851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:52:59.900000931 +0000 UTC m=+464.711348329" watchObservedRunningTime="2026-07-27 21:52:59.901958966 +0000 UTC m=+464.713306338"
Jul 27 21:53:10.888770 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:10.888732    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"
Jul 27 21:53:11.281088 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:11.281000    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"]
Jul 27 21:53:11.281281 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:11.281200    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q" podUID="7986e592-cb59-4d56-aca9-d2f1a7fd426a" containerName="limitador" containerID="cri-o://8b21c850c3efa4af5bbbfb69124821f97695482064066ed9575a8620d7c8a065" gracePeriod=30
Jul 27 21:53:11.937975 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:11.937944    2549 generic.go:356] "Generic (PLEG): container finished" podID="7986e592-cb59-4d56-aca9-d2f1a7fd426a" containerID="8b21c850c3efa4af5bbbfb69124821f97695482064066ed9575a8620d7c8a065" exitCode=0
Jul 27 21:53:11.937975 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:11.937979    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q" event={"ID":"7986e592-cb59-4d56-aca9-d2f1a7fd426a","Type":"ContainerDied","Data":"8b21c850c3efa4af5bbbfb69124821f97695482064066ed9575a8620d7c8a065"}
Jul 27 21:53:12.230095 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:12.230053    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"
Jul 27 21:53:12.335646 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:12.335616    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7986e592-cb59-4d56-aca9-d2f1a7fd426a-config-file\") pod \"7986e592-cb59-4d56-aca9-d2f1a7fd426a\" (UID: \"7986e592-cb59-4d56-aca9-d2f1a7fd426a\") "
Jul 27 21:53:12.335834 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:12.335666    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4448\" (UniqueName: \"kubernetes.io/projected/7986e592-cb59-4d56-aca9-d2f1a7fd426a-kube-api-access-d4448\") pod \"7986e592-cb59-4d56-aca9-d2f1a7fd426a\" (UID: \"7986e592-cb59-4d56-aca9-d2f1a7fd426a\") "
Jul 27 21:53:12.336035 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:12.336008    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7986e592-cb59-4d56-aca9-d2f1a7fd426a-config-file" (OuterVolumeSpecName: "config-file") pod "7986e592-cb59-4d56-aca9-d2f1a7fd426a" (UID: "7986e592-cb59-4d56-aca9-d2f1a7fd426a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:53:12.337881 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:12.337855    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7986e592-cb59-4d56-aca9-d2f1a7fd426a-kube-api-access-d4448" (OuterVolumeSpecName: "kube-api-access-d4448") pod "7986e592-cb59-4d56-aca9-d2f1a7fd426a" (UID: "7986e592-cb59-4d56-aca9-d2f1a7fd426a"). InnerVolumeSpecName "kube-api-access-d4448". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:53:12.437015 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:12.436975    2549 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7986e592-cb59-4d56-aca9-d2f1a7fd426a-config-file\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:53:12.437015 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:12.437009    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4448\" (UniqueName: \"kubernetes.io/projected/7986e592-cb59-4d56-aca9-d2f1a7fd426a-kube-api-access-d4448\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:53:12.943768 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:12.943722    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"
Jul 27 21:53:12.943768 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:12.943747    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5ss9q" event={"ID":"7986e592-cb59-4d56-aca9-d2f1a7fd426a","Type":"ContainerDied","Data":"ee3e92465486273272674dbdd0f245962c8492719a963ff9990df72b8700ad54"}
Jul 27 21:53:12.944237 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:12.943809    2549 scope.go:117] "RemoveContainer" containerID="8b21c850c3efa4af5bbbfb69124821f97695482064066ed9575a8620d7c8a065"
Jul 27 21:53:12.966331 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:12.966306    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"]
Jul 27 21:53:12.969671 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:12.969650    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5ss9q"]
Jul 27 21:53:13.700204 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:13.700174    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7986e592-cb59-4d56-aca9-d2f1a7fd426a" path="/var/lib/kubelet/pods/7986e592-cb59-4d56-aca9-d2f1a7fd426a/volumes"
Jul 27 21:53:15.701205 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.701176    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc"]
Jul 27 21:53:15.701559 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.701482    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7986e592-cb59-4d56-aca9-d2f1a7fd426a" containerName="limitador"
Jul 27 21:53:15.701559 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.701494    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7986e592-cb59-4d56-aca9-d2f1a7fd426a" containerName="limitador"
Jul 27 21:53:15.701559 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.701558    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="7986e592-cb59-4d56-aca9-d2f1a7fd426a" containerName="limitador"
Jul 27 21:53:15.705984 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.705967    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc"
Jul 27 21:53:15.713847 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.713770    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 27 21:53:15.714294 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.714277    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 21:53:15.714294 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.714290    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:53:15.714531 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.714515    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2hxdg\""
Jul 27 21:53:15.716544 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.716521    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc"]
Jul 27 21:53:15.864732 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.864701    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5986b3d3-e094-44be-a406-7651490ff78c-data\") pod \"postgres-69fb4594cf-lbsdc\" (UID: \"5986b3d3-e094-44be-a406-7651490ff78c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc"
Jul 27 21:53:15.864920 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.864817    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnsdp\" (UniqueName: \"kubernetes.io/projected/5986b3d3-e094-44be-a406-7651490ff78c-kube-api-access-tnsdp\") pod \"postgres-69fb4594cf-lbsdc\" (UID: \"5986b3d3-e094-44be-a406-7651490ff78c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc"
Jul 27 21:53:15.965835 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.965804    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnsdp\" (UniqueName: \"kubernetes.io/projected/5986b3d3-e094-44be-a406-7651490ff78c-kube-api-access-tnsdp\") pod \"postgres-69fb4594cf-lbsdc\" (UID: \"5986b3d3-e094-44be-a406-7651490ff78c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc"
Jul 27 21:53:15.966056 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.965863    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5986b3d3-e094-44be-a406-7651490ff78c-data\") pod \"postgres-69fb4594cf-lbsdc\" (UID: \"5986b3d3-e094-44be-a406-7651490ff78c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc"
Jul 27 21:53:15.966294 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.966273    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5986b3d3-e094-44be-a406-7651490ff78c-data\") pod \"postgres-69fb4594cf-lbsdc\" (UID: \"5986b3d3-e094-44be-a406-7651490ff78c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc"
Jul 27 21:53:15.983739 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:15.983706    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnsdp\" (UniqueName: \"kubernetes.io/projected/5986b3d3-e094-44be-a406-7651490ff78c-kube-api-access-tnsdp\") pod \"postgres-69fb4594cf-lbsdc\" (UID: \"5986b3d3-e094-44be-a406-7651490ff78c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc"
Jul 27 21:53:16.016664 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:16.016615    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc"
Jul 27 21:53:16.151581 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:16.151433    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc"]
Jul 27 21:53:16.968755 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:16.968717    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc" event={"ID":"5986b3d3-e094-44be-a406-7651490ff78c","Type":"ContainerStarted","Data":"88e95b7e38b218cbdf1493f6077bdb447b86421751ef23878d2e8bec5ef30876"}
Jul 27 21:53:21.993703 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:21.993664    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc" event={"ID":"5986b3d3-e094-44be-a406-7651490ff78c","Type":"ContainerStarted","Data":"ca7b236b3934fc934beb51b64b4337784eb641384e4acbd312cde176a39b160e"}
Jul 27 21:53:21.994178 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:21.993867    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc"
Jul 27 21:53:22.010987 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:22.010936    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc" podStartSLOduration=2.134597893 podStartE2EDuration="7.010920629s" totalImagesPullingTime="4.876322736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:53:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:53:22.009620774 +0000 UTC m=+486.820968146" watchObservedRunningTime="2026-07-27 21:53:22.010920629 +0000 UTC m=+486.822268000"
Jul 27 21:53:28.027271 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:28.027241    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbsdc"
Jul 27 21:53:34.782202 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:34.782167    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9f96c6d74-h9w26"]
Jul 27 21:53:34.801597 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:34.801570    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9f96c6d74-h9w26"]
Jul 27 21:53:34.801742 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:34.801691    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9f96c6d74-h9w26"
Jul 27 21:53:34.804609 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:34.804574    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d78mf\""
Jul 27 21:53:34.804756 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:34.804673    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 21:53:34.805861 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:34.805834    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 21:53:34.821414 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:34.821385    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95b3fb16-af3f-479f-a961-0a35be09541f-webhook-cert\") pod \"maas-controller-9f96c6d74-h9w26\" (UID: \"95b3fb16-af3f-479f-a961-0a35be09541f\") " pod="opendatahub/maas-controller-9f96c6d74-h9w26"
Jul 27 21:53:34.821595 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:34.821571    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg59g\" (UniqueName: \"kubernetes.io/projected/95b3fb16-af3f-479f-a961-0a35be09541f-kube-api-access-jg59g\") pod \"maas-controller-9f96c6d74-h9w26\" (UID: \"95b3fb16-af3f-479f-a961-0a35be09541f\") " pod="opendatahub/maas-controller-9f96c6d74-h9w26"
Jul 27 21:53:34.922796 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:34.922761    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg59g\" (UniqueName: \"kubernetes.io/projected/95b3fb16-af3f-479f-a961-0a35be09541f-kube-api-access-jg59g\") pod \"maas-controller-9f96c6d74-h9w26\" (UID: \"95b3fb16-af3f-479f-a961-0a35be09541f\") " pod="opendatahub/maas-controller-9f96c6d74-h9w26"
Jul 27 21:53:34.922796 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:34.922803    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95b3fb16-af3f-479f-a961-0a35be09541f-webhook-cert\") pod \"maas-controller-9f96c6d74-h9w26\" (UID: \"95b3fb16-af3f-479f-a961-0a35be09541f\") " pod="opendatahub/maas-controller-9f96c6d74-h9w26"
Jul 27 21:53:34.923060 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:53:34.922954    2549 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 27 21:53:34.923060 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:53:34.923013    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b3fb16-af3f-479f-a961-0a35be09541f-webhook-cert podName:95b3fb16-af3f-479f-a961-0a35be09541f nodeName:}" failed. No retries permitted until 2026-07-27 21:53:35.422991937 +0000 UTC m=+500.234339287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/95b3fb16-af3f-479f-a961-0a35be09541f-webhook-cert") pod "maas-controller-9f96c6d74-h9w26" (UID: "95b3fb16-af3f-479f-a961-0a35be09541f") : secret "maas-controller-webhook-cert" not found
Jul 27 21:53:34.932761 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:34.932733    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg59g\" (UniqueName: \"kubernetes.io/projected/95b3fb16-af3f-479f-a961-0a35be09541f-kube-api-access-jg59g\") pod \"maas-controller-9f96c6d74-h9w26\" (UID: \"95b3fb16-af3f-479f-a961-0a35be09541f\") " pod="opendatahub/maas-controller-9f96c6d74-h9w26"
Jul 27 21:53:34.954765 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:34.954737    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8599cc5679-xcn8h"]
Jul 27 21:53:34.957266 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:34.957215    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8599cc5679-xcn8h"
Jul 27 21:53:34.968779 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:34.968744    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8599cc5679-xcn8h"]
Jul 27 21:53:35.023301 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:35.023269    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsbbq\" (UniqueName: \"kubernetes.io/projected/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-kube-api-access-hsbbq\") pod \"maas-controller-8599cc5679-xcn8h\" (UID: \"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8\") " pod="opendatahub/maas-controller-8599cc5679-xcn8h"
Jul 27 21:53:35.023301 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:35.023307    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-webhook-cert\") pod \"maas-controller-8599cc5679-xcn8h\" (UID: \"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8\") " pod="opendatahub/maas-controller-8599cc5679-xcn8h"
Jul 27 21:53:35.124398 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:35.124359    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsbbq\" (UniqueName: \"kubernetes.io/projected/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-kube-api-access-hsbbq\") pod \"maas-controller-8599cc5679-xcn8h\" (UID: \"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8\") " pod="opendatahub/maas-controller-8599cc5679-xcn8h"
Jul 27 21:53:35.124578 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:35.124407    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-webhook-cert\") pod \"maas-controller-8599cc5679-xcn8h\" (UID: \"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8\") " pod="opendatahub/maas-controller-8599cc5679-xcn8h"
Jul 27 21:53:35.124578 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:53:35.124562    2549 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 27 21:53:35.124670 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:53:35.124615    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-webhook-cert podName:38f31ee7-0e9c-429c-b4f1-77d6ea6589f8 nodeName:}" failed. No retries permitted until 2026-07-27 21:53:35.624600604 +0000 UTC m=+500.435947954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-webhook-cert") pod "maas-controller-8599cc5679-xcn8h" (UID: "38f31ee7-0e9c-429c-b4f1-77d6ea6589f8") : secret "maas-controller-webhook-cert" not found
Jul 27 21:53:35.133280 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:35.133255    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsbbq\" (UniqueName: \"kubernetes.io/projected/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-kube-api-access-hsbbq\") pod \"maas-controller-8599cc5679-xcn8h\" (UID: \"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8\") " pod="opendatahub/maas-controller-8599cc5679-xcn8h"
Jul 27 21:53:35.427330 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:35.427213    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95b3fb16-af3f-479f-a961-0a35be09541f-webhook-cert\") pod \"maas-controller-9f96c6d74-h9w26\" (UID: \"95b3fb16-af3f-479f-a961-0a35be09541f\") " pod="opendatahub/maas-controller-9f96c6d74-h9w26"
Jul 27 21:53:35.429755 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:35.429736    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95b3fb16-af3f-479f-a961-0a35be09541f-webhook-cert\") pod \"maas-controller-9f96c6d74-h9w26\" (UID: \"95b3fb16-af3f-479f-a961-0a35be09541f\") " pod="opendatahub/maas-controller-9f96c6d74-h9w26"
Jul 27 21:53:35.628291 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:35.628249    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-webhook-cert\") pod \"maas-controller-8599cc5679-xcn8h\" (UID: \"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8\") " pod="opendatahub/maas-controller-8599cc5679-xcn8h"
Jul 27 21:53:35.630861 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:35.630842    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-webhook-cert\") pod \"maas-controller-8599cc5679-xcn8h\" (UID: \"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8\") " pod="opendatahub/maas-controller-8599cc5679-xcn8h"
Jul 27 21:53:35.712835 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:35.712754    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9f96c6d74-h9w26"
Jul 27 21:53:35.847325 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:35.847301    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9f96c6d74-h9w26"]
Jul 27 21:53:35.849440 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:53:35.849410    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b3fb16_af3f_479f_a961_0a35be09541f.slice/crio-4a807e16a70e2691d2ed7304ca2974de19790cc93d0568b82f688964f051a75f WatchSource:0}: Error finding container 4a807e16a70e2691d2ed7304ca2974de19790cc93d0568b82f688964f051a75f: Status 404 returned error can't find the container with id 4a807e16a70e2691d2ed7304ca2974de19790cc93d0568b82f688964f051a75f
Jul 27 21:53:35.867964 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:35.867938    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8599cc5679-xcn8h"
Jul 27 21:53:36.009550 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:36.009450    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8599cc5679-xcn8h"]
Jul 27 21:53:36.016143 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:53:36.016097    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f31ee7_0e9c_429c_b4f1_77d6ea6589f8.slice/crio-e77037f89159a42ddc4d8e66e5ac4b77718b20c52efb78ac5e2551faefd2c417 WatchSource:0}: Error finding container e77037f89159a42ddc4d8e66e5ac4b77718b20c52efb78ac5e2551faefd2c417: Status 404 returned error can't find the container with id e77037f89159a42ddc4d8e66e5ac4b77718b20c52efb78ac5e2551faefd2c417
Jul 27 21:53:36.057614 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:36.057575    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9f96c6d74-h9w26" event={"ID":"95b3fb16-af3f-479f-a961-0a35be09541f","Type":"ContainerStarted","Data":"4a807e16a70e2691d2ed7304ca2974de19790cc93d0568b82f688964f051a75f"}
Jul 27 21:53:36.058773 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:36.058746    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8599cc5679-xcn8h" event={"ID":"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8","Type":"ContainerStarted","Data":"e77037f89159a42ddc4d8e66e5ac4b77718b20c52efb78ac5e2551faefd2c417"}
Jul 27 21:53:39.076448 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:39.076413    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9f96c6d74-h9w26" event={"ID":"95b3fb16-af3f-479f-a961-0a35be09541f","Type":"ContainerStarted","Data":"6c5360e968ab2078e1ac5c6af1b27c9dfd14c609aded1056602e8c2e540ad1f6"}
Jul 27 21:53:39.076916 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:39.076481    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9f96c6d74-h9w26"
Jul 27 21:53:39.077989 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:39.077964    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8599cc5679-xcn8h" event={"ID":"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8","Type":"ContainerStarted","Data":"ae463c425a8d1f2e7e9bd0e40b5caaf39c1ad917966d3d56db2e63c79a3cd341"}
Jul 27 21:53:39.078113 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:39.078025    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8599cc5679-xcn8h"
Jul 27 21:53:39.098042 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:39.097992    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9f96c6d74-h9w26" podStartSLOduration=2.668893887 podStartE2EDuration="5.097976692s" totalImagesPullingTime="2.429082805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:53:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:53:39.097036494 +0000 UTC m=+503.908383863" watchObservedRunningTime="2026-07-27 21:53:39.097976692 +0000 UTC m=+503.909324066"
Jul 27 21:53:39.115643 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:39.115588    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8599cc5679-xcn8h" podStartSLOduration=2.843343007 podStartE2EDuration="5.11556885s" totalImagesPullingTime="2.272225843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:53:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:53:39.115298757 +0000 UTC m=+503.926646132" watchObservedRunningTime="2026-07-27 21:53:39.11556885 +0000 UTC m=+503.926916223"
Jul 27 21:53:50.084641 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:50.084609    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9f96c6d74-h9w26"
Jul 27 21:53:50.085191 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:50.084692    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8599cc5679-xcn8h"
Jul 27 21:53:50.149703 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:50.149660    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9f96c6d74-h9w26"]
Jul 27 21:53:50.149952 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:50.149900    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-9f96c6d74-h9w26" podUID="95b3fb16-af3f-479f-a961-0a35be09541f" containerName="manager" containerID="cri-o://6c5360e968ab2078e1ac5c6af1b27c9dfd14c609aded1056602e8c2e540ad1f6" gracePeriod=30
Jul 27 21:53:50.398903 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:50.398878    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9f96c6d74-h9w26"
Jul 27 21:53:50.441858 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:50.441825    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95b3fb16-af3f-479f-a961-0a35be09541f-webhook-cert\") pod \"95b3fb16-af3f-479f-a961-0a35be09541f\" (UID: \"95b3fb16-af3f-479f-a961-0a35be09541f\") "
Jul 27 21:53:50.442045 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:50.441865    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg59g\" (UniqueName: \"kubernetes.io/projected/95b3fb16-af3f-479f-a961-0a35be09541f-kube-api-access-jg59g\") pod \"95b3fb16-af3f-479f-a961-0a35be09541f\" (UID: \"95b3fb16-af3f-479f-a961-0a35be09541f\") "
Jul 27 21:53:50.444127 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:50.444099    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b3fb16-af3f-479f-a961-0a35be09541f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "95b3fb16-af3f-479f-a961-0a35be09541f" (UID: "95b3fb16-af3f-479f-a961-0a35be09541f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:53:50.444259 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:50.444149    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b3fb16-af3f-479f-a961-0a35be09541f-kube-api-access-jg59g" (OuterVolumeSpecName: "kube-api-access-jg59g") pod "95b3fb16-af3f-479f-a961-0a35be09541f" (UID: "95b3fb16-af3f-479f-a961-0a35be09541f"). InnerVolumeSpecName "kube-api-access-jg59g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:53:50.542503 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:50.542467    2549 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95b3fb16-af3f-479f-a961-0a35be09541f-webhook-cert\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:53:50.542503 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:50.542500    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg59g\" (UniqueName: \"kubernetes.io/projected/95b3fb16-af3f-479f-a961-0a35be09541f-kube-api-access-jg59g\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:53:51.135093 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:51.135051    2549 generic.go:356] "Generic (PLEG): container finished" podID="95b3fb16-af3f-479f-a961-0a35be09541f" containerID="6c5360e968ab2078e1ac5c6af1b27c9dfd14c609aded1056602e8c2e540ad1f6" exitCode=0
Jul 27 21:53:51.135567 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:51.135108    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9f96c6d74-h9w26"
Jul 27 21:53:51.135567 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:51.135142    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9f96c6d74-h9w26" event={"ID":"95b3fb16-af3f-479f-a961-0a35be09541f","Type":"ContainerDied","Data":"6c5360e968ab2078e1ac5c6af1b27c9dfd14c609aded1056602e8c2e540ad1f6"}
Jul 27 21:53:51.135567 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:51.135202    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9f96c6d74-h9w26" event={"ID":"95b3fb16-af3f-479f-a961-0a35be09541f","Type":"ContainerDied","Data":"4a807e16a70e2691d2ed7304ca2974de19790cc93d0568b82f688964f051a75f"}
Jul 27 21:53:51.135567 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:51.135256    2549 scope.go:117] "RemoveContainer" containerID="6c5360e968ab2078e1ac5c6af1b27c9dfd14c609aded1056602e8c2e540ad1f6"
Jul 27 21:53:51.144649 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:51.144632    2549 scope.go:117] "RemoveContainer" containerID="6c5360e968ab2078e1ac5c6af1b27c9dfd14c609aded1056602e8c2e540ad1f6"
Jul 27 21:53:51.144943 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:53:51.144925    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5360e968ab2078e1ac5c6af1b27c9dfd14c609aded1056602e8c2e540ad1f6\": container with ID starting with 6c5360e968ab2078e1ac5c6af1b27c9dfd14c609aded1056602e8c2e540ad1f6 not found: ID does not exist" containerID="6c5360e968ab2078e1ac5c6af1b27c9dfd14c609aded1056602e8c2e540ad1f6"
Jul 27 21:53:51.144990 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:51.144953    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5360e968ab2078e1ac5c6af1b27c9dfd14c609aded1056602e8c2e540ad1f6"} err="failed to get container status \"6c5360e968ab2078e1ac5c6af1b27c9dfd14c609aded1056602e8c2e540ad1f6\": rpc error: code = NotFound desc = could not find container \"6c5360e968ab2078e1ac5c6af1b27c9dfd14c609aded1056602e8c2e540ad1f6\": container with ID starting with 6c5360e968ab2078e1ac5c6af1b27c9dfd14c609aded1056602e8c2e540ad1f6 not found: ID does not exist"
Jul 27 21:53:51.162747 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:51.162709    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9f96c6d74-h9w26"]
Jul 27 21:53:51.170000 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:51.169974    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9f96c6d74-h9w26"]
Jul 27 21:53:51.700895 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:53:51.700858    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95b3fb16-af3f-479f-a961-0a35be09541f" path="/var/lib/kubelet/pods/95b3fb16-af3f-479f-a961-0a35be09541f/volumes"
Jul 27 21:54:11.922475 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:11.922438    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"]
Jul 27 21:54:11.922871 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:11.922793    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95b3fb16-af3f-479f-a961-0a35be09541f" containerName="manager"
Jul 27 21:54:11.922871 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:11.922803    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="95b3fb16-af3f-479f-a961-0a35be09541f" containerName="manager"
Jul 27 21:54:11.922871 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:11.922869    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="95b3fb16-af3f-479f-a961-0a35be09541f" containerName="manager"
Jul 27 21:54:11.930926 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:11.930905    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"
Jul 27 21:54:11.935594 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:11.935564    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 27 21:54:11.935768 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:11.935743    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ncj2z\""
Jul 27 21:54:11.937960 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:11.937929    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"]
Jul 27 21:54:12.029628 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.029585    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d76d30a-f5e8-47fd-a43c-5f787e777136-maas-api-tls\") pod \"maas-api-689cf98ffb-th6lm\" (UID: \"6d76d30a-f5e8-47fd-a43c-5f787e777136\") " pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"
Jul 27 21:54:12.029810 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.029639    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d76d30a-f5e8-47fd-a43c-5f787e777136-openshift-service-ca\") pod \"maas-api-689cf98ffb-th6lm\" (UID: \"6d76d30a-f5e8-47fd-a43c-5f787e777136\") " pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"
Jul 27 21:54:12.029810 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.029698    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd8gx\" (UniqueName: \"kubernetes.io/projected/6d76d30a-f5e8-47fd-a43c-5f787e777136-kube-api-access-qd8gx\") pod \"maas-api-689cf98ffb-th6lm\" (UID: \"6d76d30a-f5e8-47fd-a43c-5f787e777136\") " pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"
Jul 27 21:54:12.131013 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.130970    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d76d30a-f5e8-47fd-a43c-5f787e777136-maas-api-tls\") pod \"maas-api-689cf98ffb-th6lm\" (UID: \"6d76d30a-f5e8-47fd-a43c-5f787e777136\") " pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"
Jul 27 21:54:12.131190 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.131047    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d76d30a-f5e8-47fd-a43c-5f787e777136-openshift-service-ca\") pod \"maas-api-689cf98ffb-th6lm\" (UID: \"6d76d30a-f5e8-47fd-a43c-5f787e777136\") " pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"
Jul 27 21:54:12.131190 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.131092    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd8gx\" (UniqueName: \"kubernetes.io/projected/6d76d30a-f5e8-47fd-a43c-5f787e777136-kube-api-access-qd8gx\") pod \"maas-api-689cf98ffb-th6lm\" (UID: \"6d76d30a-f5e8-47fd-a43c-5f787e777136\") " pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"
Jul 27 21:54:12.131754 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.131694    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d76d30a-f5e8-47fd-a43c-5f787e777136-openshift-service-ca\") pod \"maas-api-689cf98ffb-th6lm\" (UID: \"6d76d30a-f5e8-47fd-a43c-5f787e777136\") " pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"
Jul 27 21:54:12.133602 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.133581    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d76d30a-f5e8-47fd-a43c-5f787e777136-maas-api-tls\") pod \"maas-api-689cf98ffb-th6lm\" (UID: \"6d76d30a-f5e8-47fd-a43c-5f787e777136\") " pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"
Jul 27 21:54:12.145270 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.145241    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd8gx\" (UniqueName: \"kubernetes.io/projected/6d76d30a-f5e8-47fd-a43c-5f787e777136-kube-api-access-qd8gx\") pod \"maas-api-689cf98ffb-th6lm\" (UID: \"6d76d30a-f5e8-47fd-a43c-5f787e777136\") " pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"
Jul 27 21:54:12.241843 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.241760    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"
Jul 27 21:54:12.386594 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.386559    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"]
Jul 27 21:54:12.390760 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:54:12.390726    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d76d30a_f5e8_47fd_a43c_5f787e777136.slice/crio-21ffe3389c9725e4a85b92a29a8cd2f786fa9536211cc34310952d47f9aaee57 WatchSource:0}: Error finding container 21ffe3389c9725e4a85b92a29a8cd2f786fa9536211cc34310952d47f9aaee57: Status 404 returned error can't find the container with id 21ffe3389c9725e4a85b92a29a8cd2f786fa9536211cc34310952d47f9aaee57
Jul 27 21:54:12.507795 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.507704    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg"]
Jul 27 21:54:12.514750 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.514716    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg"
Jul 27 21:54:12.517805 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.517770    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 27 21:54:12.522716 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.522631    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg"]
Jul 27 21:54:12.534746 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.534724    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-p8vvq"]
Jul 27 21:54:12.538741 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.538718    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-p8vvq"
Jul 27 21:54:12.541446 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.541420    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-65kpj\""
Jul 27 21:54:12.550838 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.550811    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-p8vvq"]
Jul 27 21:54:12.635516 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.635480    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd637d6-21ac-438c-be0c-6f2e856aea55-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-p8vvq\" (UID: \"1bd637d6-21ac-438c-be0c-6f2e856aea55\") " pod="openshift-ingress/payload-processing-5cc6cd444b-p8vvq"
Jul 27 21:54:12.635699 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.635553    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjwhn\" (UniqueName: \"kubernetes.io/projected/1bd637d6-21ac-438c-be0c-6f2e856aea55-kube-api-access-mjwhn\") pod \"payload-processing-5cc6cd444b-p8vvq\" (UID: \"1bd637d6-21ac-438c-be0c-6f2e856aea55\") " pod="openshift-ingress/payload-processing-5cc6cd444b-p8vvq"
Jul 27 21:54:12.635699 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.635614    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8cjf\" (UniqueName: \"kubernetes.io/projected/e6cb51f2-feec-4aeb-bb85-216d1e2208ea-kube-api-access-g8cjf\") pod \"payload-pre-processing-687dbb6cf6-5tfmg\" (UID: \"e6cb51f2-feec-4aeb-bb85-216d1e2208ea\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg"
Jul 27 21:54:12.635699 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.635644    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6cb51f2-feec-4aeb-bb85-216d1e2208ea-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-5tfmg\" (UID: \"e6cb51f2-feec-4aeb-bb85-216d1e2208ea\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg"
Jul 27 21:54:12.736242 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.736195    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8cjf\" (UniqueName: \"kubernetes.io/projected/e6cb51f2-feec-4aeb-bb85-216d1e2208ea-kube-api-access-g8cjf\") pod \"payload-pre-processing-687dbb6cf6-5tfmg\" (UID: \"e6cb51f2-feec-4aeb-bb85-216d1e2208ea\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg"
Jul 27 21:54:12.736414 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.736271    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6cb51f2-feec-4aeb-bb85-216d1e2208ea-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-5tfmg\" (UID: \"e6cb51f2-feec-4aeb-bb85-216d1e2208ea\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg"
Jul 27 21:54:12.736414 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.736330    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd637d6-21ac-438c-be0c-6f2e856aea55-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-p8vvq\" (UID: \"1bd637d6-21ac-438c-be0c-6f2e856aea55\") " pod="openshift-ingress/payload-processing-5cc6cd444b-p8vvq"
Jul 27 21:54:12.736414 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.736378    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjwhn\" (UniqueName: \"kubernetes.io/projected/1bd637d6-21ac-438c-be0c-6f2e856aea55-kube-api-access-mjwhn\") pod \"payload-processing-5cc6cd444b-p8vvq\" (UID: \"1bd637d6-21ac-438c-be0c-6f2e856aea55\") " pod="openshift-ingress/payload-processing-5cc6cd444b-p8vvq"
Jul 27 21:54:12.737034 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.737016    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd637d6-21ac-438c-be0c-6f2e856aea55-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-p8vvq\" (UID: \"1bd637d6-21ac-438c-be0c-6f2e856aea55\") " pod="openshift-ingress/payload-processing-5cc6cd444b-p8vvq"
Jul 27 21:54:12.737071 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.737023    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6cb51f2-feec-4aeb-bb85-216d1e2208ea-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-5tfmg\" (UID: \"e6cb51f2-feec-4aeb-bb85-216d1e2208ea\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg"
Jul 27 21:54:12.744774 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.744743    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjwhn\" (UniqueName: \"kubernetes.io/projected/1bd637d6-21ac-438c-be0c-6f2e856aea55-kube-api-access-mjwhn\") pod \"payload-processing-5cc6cd444b-p8vvq\" (UID: \"1bd637d6-21ac-438c-be0c-6f2e856aea55\") " pod="openshift-ingress/payload-processing-5cc6cd444b-p8vvq"
Jul 27 21:54:12.744904 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.744802    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8cjf\" (UniqueName: \"kubernetes.io/projected/e6cb51f2-feec-4aeb-bb85-216d1e2208ea-kube-api-access-g8cjf\") pod \"payload-pre-processing-687dbb6cf6-5tfmg\" (UID: \"e6cb51f2-feec-4aeb-bb85-216d1e2208ea\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg"
Jul 27 21:54:12.828056 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.827967    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg"
Jul 27 21:54:12.848720 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.848693    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-p8vvq"
Jul 27 21:54:12.983993 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:12.983968    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg"]
Jul 27 21:54:12.986472 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:54:12.986432    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6cb51f2_feec_4aeb_bb85_216d1e2208ea.slice/crio-6f11ba48c06d290a7c893fbd13b496e99f259dffb2e72eda1109f4752e74acdc WatchSource:0}: Error finding container 6f11ba48c06d290a7c893fbd13b496e99f259dffb2e72eda1109f4752e74acdc: Status 404 returned error can't find the container with id 6f11ba48c06d290a7c893fbd13b496e99f259dffb2e72eda1109f4752e74acdc
Jul 27 21:54:13.020742 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:13.020713    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-p8vvq"]
Jul 27 21:54:13.024129 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:54:13.024091    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd637d6_21ac_438c_be0c_6f2e856aea55.slice/crio-598340b24c64a7167635e13f2e9e7878e00f14abfbe3710f302d6b0bbaca82fa WatchSource:0}: Error finding container 598340b24c64a7167635e13f2e9e7878e00f14abfbe3710f302d6b0bbaca82fa: Status 404 returned error can't find the container with id 598340b24c64a7167635e13f2e9e7878e00f14abfbe3710f302d6b0bbaca82fa
Jul 27 21:54:13.230581 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:13.230546    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg" event={"ID":"e6cb51f2-feec-4aeb-bb85-216d1e2208ea","Type":"ContainerStarted","Data":"6f11ba48c06d290a7c893fbd13b496e99f259dffb2e72eda1109f4752e74acdc"}
Jul 27 21:54:13.231697 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:13.231666    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-p8vvq" event={"ID":"1bd637d6-21ac-438c-be0c-6f2e856aea55","Type":"ContainerStarted","Data":"598340b24c64a7167635e13f2e9e7878e00f14abfbe3710f302d6b0bbaca82fa"}
Jul 27 21:54:13.232735 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:13.232712    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm" event={"ID":"6d76d30a-f5e8-47fd-a43c-5f787e777136","Type":"ContainerStarted","Data":"21ffe3389c9725e4a85b92a29a8cd2f786fa9536211cc34310952d47f9aaee57"}
Jul 27 21:54:15.246262 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:15.246204    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm" event={"ID":"6d76d30a-f5e8-47fd-a43c-5f787e777136","Type":"ContainerStarted","Data":"d3c8703870b1c184728f2e73ac4b16859ea937d3023130a79abac611adeb2657"}
Jul 27 21:54:15.246716 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:15.246426    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"
Jul 27 21:54:15.265170 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:15.265116    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm" podStartSLOduration=2.318017473 podStartE2EDuration="4.26509519s" totalImagesPullingTime="1.947077717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:54:15.2643596 +0000 UTC m=+540.075706972" watchObservedRunningTime="2026-07-27 21:54:15.26509519 +0000 UTC m=+540.076442562"
Jul 27 21:54:16.253039 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:16.253001    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-p8vvq" event={"ID":"1bd637d6-21ac-438c-be0c-6f2e856aea55","Type":"ContainerStarted","Data":"7bf765d413e980161b7f72293d08743d58b10aea5122ab5cf3e6681c2e1f41fe"}
Jul 27 21:54:16.253505 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:16.253189    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-p8vvq"
Jul 27 21:54:16.254668 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:16.254642    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg" event={"ID":"e6cb51f2-feec-4aeb-bb85-216d1e2208ea","Type":"ContainerStarted","Data":"4d8567dfbfbdd00d0c508ee0fd941865d126cd49894bb847aa78681d84758d6a"}
Jul 27 21:54:16.254803 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:16.254768    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg"
Jul 27 21:54:16.272992 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:16.272945    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-p8vvq" podStartSLOduration=1.752503651 podStartE2EDuration="4.272929878s" totalImagesPullingTime="2.520426227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:54:16.271654028 +0000 UTC m=+541.083001412" watchObservedRunningTime="2026-07-27 21:54:16.272929878 +0000 UTC m=+541.084277250"
Jul 27 21:54:16.290287 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:16.290217    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg" podStartSLOduration=1.739009083 podStartE2EDuration="4.290198777s" totalImagesPullingTime="2.551189694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:54:16.289613282 +0000 UTC m=+541.100960658" watchObservedRunningTime="2026-07-27 21:54:16.290198777 +0000 UTC m=+541.101546152"
Jul 27 21:54:21.260025 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:21.259993    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-689cf98ffb-th6lm"
Jul 27 21:54:27.261760 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.261727    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-5tfmg"
Jul 27 21:54:27.262145 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.261814    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-p8vvq"
Jul 27 21:54:27.340717 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.340682    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"]
Jul 27 21:54:27.344413 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.344393    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.347476 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.347453    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 21:54:27.347613 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.347454    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 27 21:54:27.347613 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.347454    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h4jdf\""
Jul 27 21:54:27.347613 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.347454    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 21:54:27.353588 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.353567    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"]
Jul 27 21:54:27.470097 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.470061    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93b987c8-fb07-4985-b4b6-724842c11813-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.470312 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.470112    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b987c8-fb07-4985-b4b6-724842c11813-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.470312 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.470130    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b987c8-fb07-4985-b4b6-724842c11813-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.470312 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.470149    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93b987c8-fb07-4985-b4b6-724842c11813-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.470312 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.470214    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkz47\" (UniqueName: \"kubernetes.io/projected/93b987c8-fb07-4985-b4b6-724842c11813-kube-api-access-gkz47\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.470312 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.470300    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93b987c8-fb07-4985-b4b6-724842c11813-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.571446 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.571364    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93b987c8-fb07-4985-b4b6-724842c11813-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.571446 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.571414    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b987c8-fb07-4985-b4b6-724842c11813-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.571446 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.571432    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b987c8-fb07-4985-b4b6-724842c11813-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.571446 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.571448    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93b987c8-fb07-4985-b4b6-724842c11813-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.571730 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.571576    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkz47\" (UniqueName: \"kubernetes.io/projected/93b987c8-fb07-4985-b4b6-724842c11813-kube-api-access-gkz47\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.571730 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.571634    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93b987c8-fb07-4985-b4b6-724842c11813-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.571862 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.571840    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93b987c8-fb07-4985-b4b6-724842c11813-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.571925 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.571862    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b987c8-fb07-4985-b4b6-724842c11813-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.571925 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.571908    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93b987c8-fb07-4985-b4b6-724842c11813-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.573994 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.573972    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93b987c8-fb07-4985-b4b6-724842c11813-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.574197 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.574183    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b987c8-fb07-4985-b4b6-724842c11813-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.581506 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.581484    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkz47\" (UniqueName: \"kubernetes.io/projected/93b987c8-fb07-4985-b4b6-724842c11813-kube-api-access-gkz47\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9\" (UID: \"93b987c8-fb07-4985-b4b6-724842c11813\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.656516 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.656470    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:27.805390 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:27.805367    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"]
Jul 27 21:54:27.807469 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:54:27.807436    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b987c8_fb07_4985_b4b6_724842c11813.slice/crio-aa598c179ca3483433d3748791bdc830b861758acd415033a4eb9c149e11819a WatchSource:0}: Error finding container aa598c179ca3483433d3748791bdc830b861758acd415033a4eb9c149e11819a: Status 404 returned error can't find the container with id aa598c179ca3483433d3748791bdc830b861758acd415033a4eb9c149e11819a
Jul 27 21:54:28.329316 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:28.329260    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9" event={"ID":"93b987c8-fb07-4985-b4b6-724842c11813","Type":"ContainerStarted","Data":"aa598c179ca3483433d3748791bdc830b861758acd415033a4eb9c149e11819a"}
Jul 27 21:54:29.335153 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:29.335051    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9" event={"ID":"93b987c8-fb07-4985-b4b6-724842c11813","Type":"ContainerStarted","Data":"6a5550568592c62e925eec72a24026b3ba6aa5bca5a48a06259a15639616970d"}
Jul 27 21:54:29.361827 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:29.361766    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9" podStartSLOduration=1.253622832 podStartE2EDuration="2.361750315s" totalImagesPullingTime="1.108127483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:54:29.359624749 +0000 UTC m=+554.170972148" watchObservedRunningTime="2026-07-27 21:54:29.361750315 +0000 UTC m=+554.173097688"
Jul 27 21:54:33.841136 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:33.841100    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"]
Jul 27 21:54:33.845016 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:33.844993    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:33.847806 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:33.847784    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 27 21:54:33.864930 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:33.864713    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"]
Jul 27 21:54:33.926198 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:33.926160    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5tkg\" (UniqueName: \"kubernetes.io/projected/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-kube-api-access-v5tkg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:33.926363 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:33.926207    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:33.926363 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:33.926268    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:33.926363 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:33.926286    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:33.926363 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:33.926331    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:33.926494 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:33.926369    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:34.027029 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.026993    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5tkg\" (UniqueName: \"kubernetes.io/projected/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-kube-api-access-v5tkg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:34.027247 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.027041    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:34.027247 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.027076    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:34.027247 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.027097    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:34.027247 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.027162    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:34.027247 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.027185    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:34.027625 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.027600    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:34.027699 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.027607    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:34.027699 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.027663    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:34.029556 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.029534    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:34.029989 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.029971    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:34.039906 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.039878    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5tkg\" (UniqueName: \"kubernetes.io/projected/3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff-kube-api-access-v5tkg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cjnwt\" (UID: \"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:34.162871 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.162838    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:34.301620 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.301596    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"]
Jul 27 21:54:34.303394 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:54:34.303370    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6a7cc1_8dc6_4ed7_b99a_1bf21bc40dff.slice/crio-acfe0d9655369c519dca5be9de7c712965a597926936d5e6785ba94eaaba44ce WatchSource:0}: Error finding container acfe0d9655369c519dca5be9de7c712965a597926936d5e6785ba94eaaba44ce: Status 404 returned error can't find the container with id acfe0d9655369c519dca5be9de7c712965a597926936d5e6785ba94eaaba44ce
Jul 27 21:54:34.358948 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:34.358914    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt" event={"ID":"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff","Type":"ContainerStarted","Data":"acfe0d9655369c519dca5be9de7c712965a597926936d5e6785ba94eaaba44ce"}
Jul 27 21:54:35.365472 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:35.365434    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt" event={"ID":"3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff","Type":"ContainerStarted","Data":"a8ece49f1e97d3e955d1bdf8bf01c91d143baeb7636927a279f997f0703bf3e3"}
Jul 27 21:54:35.384103 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:35.384050    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt" podStartSLOduration=2.178355668 podStartE2EDuration="2.384036039s" totalImagesPullingTime="205.680371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:54:35.382436167 +0000 UTC m=+560.193783538" watchObservedRunningTime="2026-07-27 21:54:35.384036039 +0000 UTC m=+560.195383411"
Jul 27 21:54:37.657661 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:37.657619    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:37.658043 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:37.657702    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:37.670124 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:37.670096    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:38.391407 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.391371    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9"
Jul 27 21:54:38.446209 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.446178    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"]
Jul 27 21:54:38.449641 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.449617    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.452499 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.452480    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 27 21:54:38.458284 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.458214    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"]
Jul 27 21:54:38.461583 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.461559    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/385046ce-9465-470f-9d4b-b60954cff4ea-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.461684 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.461615    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385046ce-9465-470f-9d4b-b60954cff4ea-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.461684 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.461645    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/385046ce-9465-470f-9d4b-b60954cff4ea-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.461806 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.461680    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/385046ce-9465-470f-9d4b-b60954cff4ea-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.461806 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.461761    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/385046ce-9465-470f-9d4b-b60954cff4ea-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.461806 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.461804    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dlbf\" (UniqueName: \"kubernetes.io/projected/385046ce-9465-470f-9d4b-b60954cff4ea-kube-api-access-8dlbf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.562837 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.562797    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/385046ce-9465-470f-9d4b-b60954cff4ea-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.563009 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.562857    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/385046ce-9465-470f-9d4b-b60954cff4ea-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.563009 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.562894    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dlbf\" (UniqueName: \"kubernetes.io/projected/385046ce-9465-470f-9d4b-b60954cff4ea-kube-api-access-8dlbf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.563009 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.562954    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/385046ce-9465-470f-9d4b-b60954cff4ea-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.563009 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.562996    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385046ce-9465-470f-9d4b-b60954cff4ea-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.563282 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.563026    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/385046ce-9465-470f-9d4b-b60954cff4ea-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.563353 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.563316    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/385046ce-9465-470f-9d4b-b60954cff4ea-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.563401 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.563328    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/385046ce-9465-470f-9d4b-b60954cff4ea-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.563440 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.563410    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385046ce-9465-470f-9d4b-b60954cff4ea-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.565449 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.565427    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/385046ce-9465-470f-9d4b-b60954cff4ea-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.565711 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.565690    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/385046ce-9465-470f-9d4b-b60954cff4ea-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.572598 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.572570    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dlbf\" (UniqueName: \"kubernetes.io/projected/385046ce-9465-470f-9d4b-b60954cff4ea-kube-api-access-8dlbf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr\" (UID: \"385046ce-9465-470f-9d4b-b60954cff4ea\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.761173 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.761087    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:38.902349 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:38.902324    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"]
Jul 27 21:54:38.904315 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:54:38.904282    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385046ce_9465_470f_9d4b_b60954cff4ea.slice/crio-1f360228b9e7a0eda2a87220dcc8585fa53aa0380ff3744b45a397c9e32f9076 WatchSource:0}: Error finding container 1f360228b9e7a0eda2a87220dcc8585fa53aa0380ff3744b45a397c9e32f9076: Status 404 returned error can't find the container with id 1f360228b9e7a0eda2a87220dcc8585fa53aa0380ff3744b45a397c9e32f9076
Jul 27 21:54:39.387372 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:39.387277    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr" event={"ID":"385046ce-9465-470f-9d4b-b60954cff4ea","Type":"ContainerStarted","Data":"f2b2ee6e8893b86ca944945dee5f7c1745c0ad84f7dd2709e7478939930eb39f"}
Jul 27 21:54:39.387372 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:39.387363    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr" event={"ID":"385046ce-9465-470f-9d4b-b60954cff4ea","Type":"ContainerStarted","Data":"1f360228b9e7a0eda2a87220dcc8585fa53aa0380ff3744b45a397c9e32f9076"}
Jul 27 21:54:39.406407 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:39.406357    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr" podStartSLOduration=1.198356792 podStartE2EDuration="1.406340019s" totalImagesPullingTime="207.983227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:54:39.405470708 +0000 UTC m=+564.216818107" watchObservedRunningTime="2026-07-27 21:54:39.406340019 +0000 UTC m=+564.217687391"
Jul 27 21:54:44.163333 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:44.163285    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:44.163776 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:44.163637    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:44.176147 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:44.176117    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:44.423189 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:44.423100    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cjnwt"
Jul 27 21:54:48.761693 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:48.761638    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:48.762085 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:48.761719    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:48.774483 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:48.774449    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:54:49.447643 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:54:49.447614    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr"
Jul 27 21:55:15.594237 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:55:15.594205    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 21:55:15.595875 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:55:15.595852    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 21:57:10.200341 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:10.200295    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8599cc5679-xcn8h"]
Jul 27 21:57:10.200838 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:10.200561    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8599cc5679-xcn8h" podUID="38f31ee7-0e9c-429c-b4f1-77d6ea6589f8" containerName="manager" containerID="cri-o://ae463c425a8d1f2e7e9bd0e40b5caaf39c1ad917966d3d56db2e63c79a3cd341" gracePeriod=30
Jul 27 21:57:10.455025 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:10.454964    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8599cc5679-xcn8h"
Jul 27 21:57:10.513439 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:10.513383    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsbbq\" (UniqueName: \"kubernetes.io/projected/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-kube-api-access-hsbbq\") pod \"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8\" (UID: \"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8\") "
Jul 27 21:57:10.513602 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:10.513471    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-webhook-cert\") pod \"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8\" (UID: \"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8\") "
Jul 27 21:57:10.515550 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:10.515515    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-kube-api-access-hsbbq" (OuterVolumeSpecName: "kube-api-access-hsbbq") pod "38f31ee7-0e9c-429c-b4f1-77d6ea6589f8" (UID: "38f31ee7-0e9c-429c-b4f1-77d6ea6589f8"). InnerVolumeSpecName "kube-api-access-hsbbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:57:10.515550 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:10.515543    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "38f31ee7-0e9c-429c-b4f1-77d6ea6589f8" (UID: "38f31ee7-0e9c-429c-b4f1-77d6ea6589f8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:57:10.620342 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:10.614933    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsbbq\" (UniqueName: \"kubernetes.io/projected/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-kube-api-access-hsbbq\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:57:10.620342 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:10.614969    2549 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8-webhook-cert\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:57:11.111471 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:11.111431    2549 generic.go:356] "Generic (PLEG): container finished" podID="38f31ee7-0e9c-429c-b4f1-77d6ea6589f8" containerID="ae463c425a8d1f2e7e9bd0e40b5caaf39c1ad917966d3d56db2e63c79a3cd341" exitCode=0
Jul 27 21:57:11.111679 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:11.111495    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8599cc5679-xcn8h"
Jul 27 21:57:11.111679 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:11.111514    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8599cc5679-xcn8h" event={"ID":"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8","Type":"ContainerDied","Data":"ae463c425a8d1f2e7e9bd0e40b5caaf39c1ad917966d3d56db2e63c79a3cd341"}
Jul 27 21:57:11.111679 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:11.111565    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8599cc5679-xcn8h" event={"ID":"38f31ee7-0e9c-429c-b4f1-77d6ea6589f8","Type":"ContainerDied","Data":"e77037f89159a42ddc4d8e66e5ac4b77718b20c52efb78ac5e2551faefd2c417"}
Jul 27 21:57:11.111679 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:11.111587    2549 scope.go:117] "RemoveContainer" containerID="ae463c425a8d1f2e7e9bd0e40b5caaf39c1ad917966d3d56db2e63c79a3cd341"
Jul 27 21:57:11.120783 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:11.120762    2549 scope.go:117] "RemoveContainer" containerID="ae463c425a8d1f2e7e9bd0e40b5caaf39c1ad917966d3d56db2e63c79a3cd341"
Jul 27 21:57:11.121047 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:57:11.121028    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae463c425a8d1f2e7e9bd0e40b5caaf39c1ad917966d3d56db2e63c79a3cd341\": container with ID starting with ae463c425a8d1f2e7e9bd0e40b5caaf39c1ad917966d3d56db2e63c79a3cd341 not found: ID does not exist" containerID="ae463c425a8d1f2e7e9bd0e40b5caaf39c1ad917966d3d56db2e63c79a3cd341"
Jul 27 21:57:11.121111 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:11.121060    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae463c425a8d1f2e7e9bd0e40b5caaf39c1ad917966d3d56db2e63c79a3cd341"} err="failed to get container status \"ae463c425a8d1f2e7e9bd0e40b5caaf39c1ad917966d3d56db2e63c79a3cd341\": rpc error: code = NotFound desc = could not find container \"ae463c425a8d1f2e7e9bd0e40b5caaf39c1ad917966d3d56db2e63c79a3cd341\": container with ID starting with ae463c425a8d1f2e7e9bd0e40b5caaf39c1ad917966d3d56db2e63c79a3cd341 not found: ID does not exist"
Jul 27 21:57:11.133653 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:11.133624    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8599cc5679-xcn8h"]
Jul 27 21:57:11.137728 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:11.137693    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8599cc5679-xcn8h"]
Jul 27 21:57:11.700610 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:57:11.700577    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38f31ee7-0e9c-429c-b4f1-77d6ea6589f8" path="/var/lib/kubelet/pods/38f31ee7-0e9c-429c-b4f1-77d6ea6589f8/volumes"
Jul 27 21:58:11.955009 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:11.954977    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8599cc5679-wp9b8"]
Jul 27 21:58:11.955564 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:11.955544    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f31ee7-0e9c-429c-b4f1-77d6ea6589f8" containerName="manager"
Jul 27 21:58:11.955641 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:11.955568    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f31ee7-0e9c-429c-b4f1-77d6ea6589f8" containerName="manager"
Jul 27 21:58:11.955711 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:11.955698    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="38f31ee7-0e9c-429c-b4f1-77d6ea6589f8" containerName="manager"
Jul 27 21:58:11.958599 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:11.958578    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8599cc5679-wp9b8"
Jul 27 21:58:11.962580 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:11.962553    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d78mf\""
Jul 27 21:58:11.962729 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:11.962603    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 21:58:11.962729 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:11.962563    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 21:58:11.965547 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:11.965515    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8599cc5679-wp9b8"]
Jul 27 21:58:12.062022 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:12.061987    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64204345-a13a-4641-95e1-e4c11790c446-webhook-cert\") pod \"maas-controller-8599cc5679-wp9b8\" (UID: \"64204345-a13a-4641-95e1-e4c11790c446\") " pod="opendatahub/maas-controller-8599cc5679-wp9b8"
Jul 27 21:58:12.062022 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:12.062027    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nln2v\" (UniqueName: \"kubernetes.io/projected/64204345-a13a-4641-95e1-e4c11790c446-kube-api-access-nln2v\") pod \"maas-controller-8599cc5679-wp9b8\" (UID: \"64204345-a13a-4641-95e1-e4c11790c446\") " pod="opendatahub/maas-controller-8599cc5679-wp9b8"
Jul 27 21:58:12.163295 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:12.163251    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64204345-a13a-4641-95e1-e4c11790c446-webhook-cert\") pod \"maas-controller-8599cc5679-wp9b8\" (UID: \"64204345-a13a-4641-95e1-e4c11790c446\") " pod="opendatahub/maas-controller-8599cc5679-wp9b8"
Jul 27 21:58:12.163295 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:12.163298    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nln2v\" (UniqueName: \"kubernetes.io/projected/64204345-a13a-4641-95e1-e4c11790c446-kube-api-access-nln2v\") pod \"maas-controller-8599cc5679-wp9b8\" (UID: \"64204345-a13a-4641-95e1-e4c11790c446\") " pod="opendatahub/maas-controller-8599cc5679-wp9b8"
Jul 27 21:58:12.165796 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:12.165771    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64204345-a13a-4641-95e1-e4c11790c446-webhook-cert\") pod \"maas-controller-8599cc5679-wp9b8\" (UID: \"64204345-a13a-4641-95e1-e4c11790c446\") " pod="opendatahub/maas-controller-8599cc5679-wp9b8"
Jul 27 21:58:12.173287 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:12.173262    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nln2v\" (UniqueName: \"kubernetes.io/projected/64204345-a13a-4641-95e1-e4c11790c446-kube-api-access-nln2v\") pod \"maas-controller-8599cc5679-wp9b8\" (UID: \"64204345-a13a-4641-95e1-e4c11790c446\") " pod="opendatahub/maas-controller-8599cc5679-wp9b8"
Jul 27 21:58:12.271365 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:12.271281    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8599cc5679-wp9b8"
Jul 27 21:58:12.412275 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:12.412251    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8599cc5679-wp9b8"]
Jul 27 21:58:12.414544 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:58:12.414510    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64204345_a13a_4641_95e1_e4c11790c446.slice/crio-8ab2148ae32ede89ef28fedbf3acdb0962afb3fcdfc3e029563a92b8761e305c WatchSource:0}: Error finding container 8ab2148ae32ede89ef28fedbf3acdb0962afb3fcdfc3e029563a92b8761e305c: Status 404 returned error can't find the container with id 8ab2148ae32ede89ef28fedbf3acdb0962afb3fcdfc3e029563a92b8761e305c
Jul 27 21:58:12.415810 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:12.415790    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:58:13.408648 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:13.408611    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8599cc5679-wp9b8" event={"ID":"64204345-a13a-4641-95e1-e4c11790c446","Type":"ContainerStarted","Data":"776539133e09be93333090adcb48c8e44afe2d1b4719e6c7b443165e9466682b"}
Jul 27 21:58:13.409042 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:13.408674    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8599cc5679-wp9b8" event={"ID":"64204345-a13a-4641-95e1-e4c11790c446","Type":"ContainerStarted","Data":"8ab2148ae32ede89ef28fedbf3acdb0962afb3fcdfc3e029563a92b8761e305c"}
Jul 27 21:58:13.409042 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:13.408737    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8599cc5679-wp9b8"
Jul 27 21:58:13.426642 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:13.426594    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8599cc5679-wp9b8" podStartSLOduration=1.849587816 podStartE2EDuration="2.426574207s" totalImagesPullingTime="576.986391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:58:13.424051588 +0000 UTC m=+778.235398960" watchObservedRunningTime="2026-07-27 21:58:13.426574207 +0000 UTC m=+778.237921612"
Jul 27 21:58:24.416534 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:24.416501    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8599cc5679-wp9b8"
Jul 27 21:58:29.084609 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.084563    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8599cc5679-wp9b8"]
Jul 27 21:58:29.085200 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.084822    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8599cc5679-wp9b8" podUID="64204345-a13a-4641-95e1-e4c11790c446" containerName="manager" containerID="cri-o://776539133e09be93333090adcb48c8e44afe2d1b4719e6c7b443165e9466682b" gracePeriod=30
Jul 27 21:58:29.337518 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.337454    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8599cc5679-wp9b8"
Jul 27 21:58:29.416549 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.416513    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64204345-a13a-4641-95e1-e4c11790c446-webhook-cert\") pod \"64204345-a13a-4641-95e1-e4c11790c446\" (UID: \"64204345-a13a-4641-95e1-e4c11790c446\") "
Jul 27 21:58:29.416549 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.416553    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nln2v\" (UniqueName: \"kubernetes.io/projected/64204345-a13a-4641-95e1-e4c11790c446-kube-api-access-nln2v\") pod \"64204345-a13a-4641-95e1-e4c11790c446\" (UID: \"64204345-a13a-4641-95e1-e4c11790c446\") "
Jul 27 21:58:29.418708 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.418679    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64204345-a13a-4641-95e1-e4c11790c446-kube-api-access-nln2v" (OuterVolumeSpecName: "kube-api-access-nln2v") pod "64204345-a13a-4641-95e1-e4c11790c446" (UID: "64204345-a13a-4641-95e1-e4c11790c446"). InnerVolumeSpecName "kube-api-access-nln2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:58:29.418816 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.418757    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64204345-a13a-4641-95e1-e4c11790c446-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "64204345-a13a-4641-95e1-e4c11790c446" (UID: "64204345-a13a-4641-95e1-e4c11790c446"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:58:29.486561 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.486521    2549 generic.go:356] "Generic (PLEG): container finished" podID="64204345-a13a-4641-95e1-e4c11790c446" containerID="776539133e09be93333090adcb48c8e44afe2d1b4719e6c7b443165e9466682b" exitCode=0
Jul 27 21:58:29.486724 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.486575    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8599cc5679-wp9b8"
Jul 27 21:58:29.486724 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.486593    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8599cc5679-wp9b8" event={"ID":"64204345-a13a-4641-95e1-e4c11790c446","Type":"ContainerDied","Data":"776539133e09be93333090adcb48c8e44afe2d1b4719e6c7b443165e9466682b"}
Jul 27 21:58:29.486724 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.486636    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8599cc5679-wp9b8" event={"ID":"64204345-a13a-4641-95e1-e4c11790c446","Type":"ContainerDied","Data":"8ab2148ae32ede89ef28fedbf3acdb0962afb3fcdfc3e029563a92b8761e305c"}
Jul 27 21:58:29.486724 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.486656    2549 scope.go:117] "RemoveContainer" containerID="776539133e09be93333090adcb48c8e44afe2d1b4719e6c7b443165e9466682b"
Jul 27 21:58:29.495683 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.495665    2549 scope.go:117] "RemoveContainer" containerID="776539133e09be93333090adcb48c8e44afe2d1b4719e6c7b443165e9466682b"
Jul 27 21:58:29.495942 ip-10-0-138-124 kubenswrapper[2549]: E0727 21:58:29.495925    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776539133e09be93333090adcb48c8e44afe2d1b4719e6c7b443165e9466682b\": container with ID starting with 776539133e09be93333090adcb48c8e44afe2d1b4719e6c7b443165e9466682b not found: ID does not exist" containerID="776539133e09be93333090adcb48c8e44afe2d1b4719e6c7b443165e9466682b"
Jul 27 21:58:29.495985 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.495950    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776539133e09be93333090adcb48c8e44afe2d1b4719e6c7b443165e9466682b"} err="failed to get container status \"776539133e09be93333090adcb48c8e44afe2d1b4719e6c7b443165e9466682b\": rpc error: code = NotFound desc = could not find container \"776539133e09be93333090adcb48c8e44afe2d1b4719e6c7b443165e9466682b\": container with ID starting with 776539133e09be93333090adcb48c8e44afe2d1b4719e6c7b443165e9466682b not found: ID does not exist"
Jul 27 21:58:29.507813 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.507787    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8599cc5679-wp9b8"]
Jul 27 21:58:29.513834 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.513812    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8599cc5679-wp9b8"]
Jul 27 21:58:29.517541 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.517523    2549 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64204345-a13a-4641-95e1-e4c11790c446-webhook-cert\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:58:29.517618 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.517544    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nln2v\" (UniqueName: \"kubernetes.io/projected/64204345-a13a-4641-95e1-e4c11790c446-kube-api-access-nln2v\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 21:58:29.700135 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:29.700088    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64204345-a13a-4641-95e1-e4c11790c446" path="/var/lib/kubelet/pods/64204345-a13a-4641-95e1-e4c11790c446/volumes"
Jul 27 21:58:30.364947 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.364913    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8599cc5679-t4hzv"]
Jul 27 21:58:30.365373 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.365317    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64204345-a13a-4641-95e1-e4c11790c446" containerName="manager"
Jul 27 21:58:30.365373 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.365330    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="64204345-a13a-4641-95e1-e4c11790c446" containerName="manager"
Jul 27 21:58:30.365445 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.365426    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="64204345-a13a-4641-95e1-e4c11790c446" containerName="manager"
Jul 27 21:58:30.370092 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.370066    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8599cc5679-t4hzv"
Jul 27 21:58:30.374484 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.374314    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 21:58:30.374484 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.374327    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 21:58:30.374484 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.374352    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d78mf\""
Jul 27 21:58:30.377324 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.377285    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8599cc5679-t4hzv"]
Jul 27 21:58:30.527866 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.527826    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmr7v\" (UniqueName: \"kubernetes.io/projected/beb9e0cf-fb12-40e2-a171-4a492d8debaf-kube-api-access-gmr7v\") pod \"maas-controller-8599cc5679-t4hzv\" (UID: \"beb9e0cf-fb12-40e2-a171-4a492d8debaf\") " pod="opendatahub/maas-controller-8599cc5679-t4hzv"
Jul 27 21:58:30.528033 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.527880    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/beb9e0cf-fb12-40e2-a171-4a492d8debaf-webhook-cert\") pod \"maas-controller-8599cc5679-t4hzv\" (UID: \"beb9e0cf-fb12-40e2-a171-4a492d8debaf\") " pod="opendatahub/maas-controller-8599cc5679-t4hzv"
Jul 27 21:58:30.629197 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.629110    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmr7v\" (UniqueName: \"kubernetes.io/projected/beb9e0cf-fb12-40e2-a171-4a492d8debaf-kube-api-access-gmr7v\") pod \"maas-controller-8599cc5679-t4hzv\" (UID: \"beb9e0cf-fb12-40e2-a171-4a492d8debaf\") " pod="opendatahub/maas-controller-8599cc5679-t4hzv"
Jul 27 21:58:30.629197 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.629162    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/beb9e0cf-fb12-40e2-a171-4a492d8debaf-webhook-cert\") pod \"maas-controller-8599cc5679-t4hzv\" (UID: \"beb9e0cf-fb12-40e2-a171-4a492d8debaf\") " pod="opendatahub/maas-controller-8599cc5679-t4hzv"
Jul 27 21:58:30.631753 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.631728    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/beb9e0cf-fb12-40e2-a171-4a492d8debaf-webhook-cert\") pod \"maas-controller-8599cc5679-t4hzv\" (UID: \"beb9e0cf-fb12-40e2-a171-4a492d8debaf\") " pod="opendatahub/maas-controller-8599cc5679-t4hzv"
Jul 27 21:58:30.638319 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.638293    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmr7v\" (UniqueName: \"kubernetes.io/projected/beb9e0cf-fb12-40e2-a171-4a492d8debaf-kube-api-access-gmr7v\") pod \"maas-controller-8599cc5679-t4hzv\" (UID: \"beb9e0cf-fb12-40e2-a171-4a492d8debaf\") " pod="opendatahub/maas-controller-8599cc5679-t4hzv"
Jul 27 21:58:30.684260 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.684199    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8599cc5679-t4hzv"
Jul 27 21:58:30.824252 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:30.824207    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8599cc5679-t4hzv"]
Jul 27 21:58:30.826992 ip-10-0-138-124 kubenswrapper[2549]: W0727 21:58:30.826963    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb9e0cf_fb12_40e2_a171_4a492d8debaf.slice/crio-757ea49f0e5efca6b6cc2bd5789a0039314f737866160527b1318cca202ac74b WatchSource:0}: Error finding container 757ea49f0e5efca6b6cc2bd5789a0039314f737866160527b1318cca202ac74b: Status 404 returned error can't find the container with id 757ea49f0e5efca6b6cc2bd5789a0039314f737866160527b1318cca202ac74b
Jul 27 21:58:31.498907 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:31.498815    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8599cc5679-t4hzv" event={"ID":"beb9e0cf-fb12-40e2-a171-4a492d8debaf","Type":"ContainerStarted","Data":"6965054f053cd12f6f002210927fa58c975b7814a3ea5594d17fc909e4095247"}
Jul 27 21:58:31.498907 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:31.498879    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8599cc5679-t4hzv"
Jul 27 21:58:31.498907 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:31.498896    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8599cc5679-t4hzv" event={"ID":"beb9e0cf-fb12-40e2-a171-4a492d8debaf","Type":"ContainerStarted","Data":"757ea49f0e5efca6b6cc2bd5789a0039314f737866160527b1318cca202ac74b"}
Jul 27 21:58:31.517936 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:31.517891    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8599cc5679-t4hzv" podStartSLOduration=1.153222805 podStartE2EDuration="1.517868466s" totalImagesPullingTime="364.645661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:58:31.5161598 +0000 UTC m=+796.327507184" watchObservedRunningTime="2026-07-27 21:58:31.517868466 +0000 UTC m=+796.329215860"
Jul 27 21:58:42.506438 ip-10-0-138-124 kubenswrapper[2549]: I0727 21:58:42.506362    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8599cc5679-t4hzv"
Jul 27 22:00:00.136350 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:00.136310    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753160-lpqtd"]
Jul 27 22:00:00.140019 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:00.139995    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753160-lpqtd"
Jul 27 22:00:00.142909 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:00.142888    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\""
Jul 27 22:00:00.146500 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:00.146468    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753160-lpqtd"]
Jul 27 22:00:00.176341 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:00.176319    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753160-lpqtd"
Jul 27 22:00:00.533839 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:00.533816    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753160-lpqtd"]
Jul 27 22:00:00.536321 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:00:00.536298    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94cf85e1_d268_4318_89f5_9b95af84405a.slice/crio-721bca551047326d8e9781e877456f00be62b7b7e4ddb67a8c2540ff3b245f2e WatchSource:0}: Error finding container 721bca551047326d8e9781e877456f00be62b7b7e4ddb67a8c2540ff3b245f2e: Status 404 returned error can't find the container with id 721bca551047326d8e9781e877456f00be62b7b7e4ddb67a8c2540ff3b245f2e
Jul 27 22:00:00.926689 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:00.926647    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753160-lpqtd" event={"ID":"94cf85e1-d268-4318-89f5-9b95af84405a","Type":"ContainerStarted","Data":"721bca551047326d8e9781e877456f00be62b7b7e4ddb67a8c2540ff3b245f2e"}
Jul 27 22:00:05.955595 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:05.955562    2549 generic.go:356] "Generic (PLEG): container finished" podID="94cf85e1-d268-4318-89f5-9b95af84405a" containerID="7656eef0f8d35a600332da5610bb45480065e2d5d087d27f850c5007baab786b" exitCode=0
Jul 27 22:00:05.956003 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:05.955637    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753160-lpqtd" event={"ID":"94cf85e1-d268-4318-89f5-9b95af84405a","Type":"ContainerDied","Data":"7656eef0f8d35a600332da5610bb45480065e2d5d087d27f850c5007baab786b"}
Jul 27 22:00:07.091594 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:07.091568    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753160-lpqtd"
Jul 27 22:00:07.966595 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:07.966501    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753160-lpqtd" event={"ID":"94cf85e1-d268-4318-89f5-9b95af84405a","Type":"ContainerDied","Data":"721bca551047326d8e9781e877456f00be62b7b7e4ddb67a8c2540ff3b245f2e"}
Jul 27 22:00:07.966595 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:07.966525    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753160-lpqtd"
Jul 27 22:00:07.966595 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:07.966550    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="721bca551047326d8e9781e877456f00be62b7b7e4ddb67a8c2540ff3b245f2e"
Jul 27 22:00:15.632280 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:15.632250    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:00:15.632280 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:00:15.632253    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:05:15.660953 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:05:15.660921    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:05:15.662120 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:05:15.662098    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:10:15.688464 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:10:15.688435    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:10:15.690408 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:10:15.690381    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:11:26.967240 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:26.967193    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"]
Jul 27 22:11:26.967709 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:26.967665    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94cf85e1-d268-4318-89f5-9b95af84405a" containerName="cleanup"
Jul 27 22:11:26.967709 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:26.967684    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="94cf85e1-d268-4318-89f5-9b95af84405a" containerName="cleanup"
Jul 27 22:11:26.967791 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:26.967778    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="94cf85e1-d268-4318-89f5-9b95af84405a" containerName="cleanup"
Jul 27 22:11:26.970734 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:26.970717    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:11:26.973458 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:26.973437    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a66e79c4-serving-cert\""
Jul 27 22:11:26.981376 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:26.981352    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"]
Jul 27 22:11:27.082871 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.082836    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b619819-cc49-49ea-b448-a03abe8e2a0d-openshift-service-ca\") pod \"maas-api-e2e-ait-a66e79c4-566988998-c9td6\" (UID: \"8b619819-cc49-49ea-b448-a03abe8e2a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:11:27.082871 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.082877    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rktjj\" (UniqueName: \"kubernetes.io/projected/8b619819-cc49-49ea-b448-a03abe8e2a0d-kube-api-access-rktjj\") pod \"maas-api-e2e-ait-a66e79c4-566988998-c9td6\" (UID: \"8b619819-cc49-49ea-b448-a03abe8e2a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:11:27.083112 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.082921    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b619819-cc49-49ea-b448-a03abe8e2a0d-maas-api-tls\") pod \"maas-api-e2e-ait-a66e79c4-566988998-c9td6\" (UID: \"8b619819-cc49-49ea-b448-a03abe8e2a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:11:27.183613 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.183580    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b619819-cc49-49ea-b448-a03abe8e2a0d-openshift-service-ca\") pod \"maas-api-e2e-ait-a66e79c4-566988998-c9td6\" (UID: \"8b619819-cc49-49ea-b448-a03abe8e2a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:11:27.183613 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.183615    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rktjj\" (UniqueName: \"kubernetes.io/projected/8b619819-cc49-49ea-b448-a03abe8e2a0d-kube-api-access-rktjj\") pod \"maas-api-e2e-ait-a66e79c4-566988998-c9td6\" (UID: \"8b619819-cc49-49ea-b448-a03abe8e2a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:11:27.183844 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.183642    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b619819-cc49-49ea-b448-a03abe8e2a0d-maas-api-tls\") pod \"maas-api-e2e-ait-a66e79c4-566988998-c9td6\" (UID: \"8b619819-cc49-49ea-b448-a03abe8e2a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:11:27.184387 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.184363    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b619819-cc49-49ea-b448-a03abe8e2a0d-openshift-service-ca\") pod \"maas-api-e2e-ait-a66e79c4-566988998-c9td6\" (UID: \"8b619819-cc49-49ea-b448-a03abe8e2a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:11:27.186314 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.186293    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b619819-cc49-49ea-b448-a03abe8e2a0d-maas-api-tls\") pod \"maas-api-e2e-ait-a66e79c4-566988998-c9td6\" (UID: \"8b619819-cc49-49ea-b448-a03abe8e2a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:11:27.194421 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.194390    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rktjj\" (UniqueName: \"kubernetes.io/projected/8b619819-cc49-49ea-b448-a03abe8e2a0d-kube-api-access-rktjj\") pod \"maas-api-e2e-ait-a66e79c4-566988998-c9td6\" (UID: \"8b619819-cc49-49ea-b448-a03abe8e2a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:11:27.281455 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.281375    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:11:27.448538 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.448507    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"]
Jul 27 22:11:27.450483 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:11:27.450451    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b619819_cc49_49ea_b448_a03abe8e2a0d.slice/crio-b205d2ce765d3851fd644372088d08e316d606fa4ef840eee693b3f6d66b5609 WatchSource:0}: Error finding container b205d2ce765d3851fd644372088d08e316d606fa4ef840eee693b3f6d66b5609: Status 404 returned error can't find the container with id b205d2ce765d3851fd644372088d08e316d606fa4ef840eee693b3f6d66b5609
Jul 27 22:11:27.451823 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.451803    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 22:11:27.701971 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.701940    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"]
Jul 27 22:11:27.706779 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.706757    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"
Jul 27 22:11:27.713062 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.713043    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a66e79c4\""
Jul 27 22:11:27.723835 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.723808    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"]
Jul 27 22:11:27.751189 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.751159    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"]
Jul 27 22:11:27.754781 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.754761    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"
Jul 27 22:11:27.758596 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.758579    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a66e79c4-dockercfg-b8klr\""
Jul 27 22:11:27.776257 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.776214    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"]
Jul 27 22:11:27.788706 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.788666    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd81a934-def5-4a06-aff0-58595a0194f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp\" (UID: \"cd81a934-def5-4a06-aff0-58595a0194f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"
Jul 27 22:11:27.788870 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.788727    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6xkq\" (UniqueName: \"kubernetes.io/projected/cd81a934-def5-4a06-aff0-58595a0194f7-kube-api-access-l6xkq\") pod \"payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp\" (UID: \"cd81a934-def5-4a06-aff0-58595a0194f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"
Jul 27 22:11:27.890207 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.890171    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvmhw\" (UniqueName: \"kubernetes.io/projected/5d35f38f-85a2-4e0b-8969-7757ed583e8b-kube-api-access-fvmhw\") pod \"payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b\" (UID: \"5d35f38f-85a2-4e0b-8969-7757ed583e8b\") " pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"
Jul 27 22:11:27.890354 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.890301    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd81a934-def5-4a06-aff0-58595a0194f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp\" (UID: \"cd81a934-def5-4a06-aff0-58595a0194f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"
Jul 27 22:11:27.890354 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.890328    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d35f38f-85a2-4e0b-8969-7757ed583e8b-plugins-config-volume\") pod \"payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b\" (UID: \"5d35f38f-85a2-4e0b-8969-7757ed583e8b\") " pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"
Jul 27 22:11:27.890437 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.890368    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6xkq\" (UniqueName: \"kubernetes.io/projected/cd81a934-def5-4a06-aff0-58595a0194f7-kube-api-access-l6xkq\") pod \"payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp\" (UID: \"cd81a934-def5-4a06-aff0-58595a0194f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"
Jul 27 22:11:27.890979 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.890961    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd81a934-def5-4a06-aff0-58595a0194f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp\" (UID: \"cd81a934-def5-4a06-aff0-58595a0194f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"
Jul 27 22:11:27.902250 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.902205    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6xkq\" (UniqueName: \"kubernetes.io/projected/cd81a934-def5-4a06-aff0-58595a0194f7-kube-api-access-l6xkq\") pod \"payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp\" (UID: \"cd81a934-def5-4a06-aff0-58595a0194f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"
Jul 27 22:11:27.991847 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.991756    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvmhw\" (UniqueName: \"kubernetes.io/projected/5d35f38f-85a2-4e0b-8969-7757ed583e8b-kube-api-access-fvmhw\") pod \"payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b\" (UID: \"5d35f38f-85a2-4e0b-8969-7757ed583e8b\") " pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"
Jul 27 22:11:27.991847 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.991836    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d35f38f-85a2-4e0b-8969-7757ed583e8b-plugins-config-volume\") pod \"payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b\" (UID: \"5d35f38f-85a2-4e0b-8969-7757ed583e8b\") " pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"
Jul 27 22:11:27.992767 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:27.992717    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d35f38f-85a2-4e0b-8969-7757ed583e8b-plugins-config-volume\") pod \"payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b\" (UID: \"5d35f38f-85a2-4e0b-8969-7757ed583e8b\") " pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"
Jul 27 22:11:28.003138 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:28.003105    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvmhw\" (UniqueName: \"kubernetes.io/projected/5d35f38f-85a2-4e0b-8969-7757ed583e8b-kube-api-access-fvmhw\") pod \"payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b\" (UID: \"5d35f38f-85a2-4e0b-8969-7757ed583e8b\") " pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"
Jul 27 22:11:28.016804 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:28.016771    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"
Jul 27 22:11:28.064758 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:28.064724    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"
Jul 27 22:11:28.177870 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:28.177765    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"]
Jul 27 22:11:28.178758 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:11:28.178726    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd81a934_def5_4a06_aff0_58595a0194f7.slice/crio-bda2fb1da88d8b5b00937f8471fd6b4971828fe08301d1e48ca8cd35efd81cd0 WatchSource:0}: Error finding container bda2fb1da88d8b5b00937f8471fd6b4971828fe08301d1e48ca8cd35efd81cd0: Status 404 returned error can't find the container with id bda2fb1da88d8b5b00937f8471fd6b4971828fe08301d1e48ca8cd35efd81cd0
Jul 27 22:11:28.234845 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:28.234815    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"]
Jul 27 22:11:28.236684 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:11:28.236656    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d35f38f_85a2_4e0b_8969_7757ed583e8b.slice/crio-b6730cae8b2f6756a85b4cdf86c6d262f8ffb6c59cc0f5510d8a92766aa8fa19 WatchSource:0}: Error finding container b6730cae8b2f6756a85b4cdf86c6d262f8ffb6c59cc0f5510d8a92766aa8fa19: Status 404 returned error can't find the container with id b6730cae8b2f6756a85b4cdf86c6d262f8ffb6c59cc0f5510d8a92766aa8fa19
Jul 27 22:11:28.296614 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:28.296579    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b" event={"ID":"5d35f38f-85a2-4e0b-8969-7757ed583e8b","Type":"ContainerStarted","Data":"b6730cae8b2f6756a85b4cdf86c6d262f8ffb6c59cc0f5510d8a92766aa8fa19"}
Jul 27 22:11:28.298723 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:28.298695    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp" event={"ID":"cd81a934-def5-4a06-aff0-58595a0194f7","Type":"ContainerStarted","Data":"beef416bf8a9694d74df47a776348fade79ee44425ed171541c77b90c4242cc5"}
Jul 27 22:11:28.298861 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:28.298744    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp" event={"ID":"cd81a934-def5-4a06-aff0-58595a0194f7","Type":"ContainerStarted","Data":"bda2fb1da88d8b5b00937f8471fd6b4971828fe08301d1e48ca8cd35efd81cd0"}
Jul 27 22:11:28.299301 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:28.298876    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"
Jul 27 22:11:28.300522 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:28.300489    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6" event={"ID":"8b619819-cc49-49ea-b448-a03abe8e2a0d","Type":"ContainerStarted","Data":"c07a6a9485aee0ff690f86fcf2bf8091e8d9dfd969b842871b81451b8b7bafdb"}
Jul 27 22:11:28.300615 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:28.300539    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6" event={"ID":"8b619819-cc49-49ea-b448-a03abe8e2a0d","Type":"ContainerStarted","Data":"b205d2ce765d3851fd644372088d08e316d606fa4ef840eee693b3f6d66b5609"}
Jul 27 22:11:28.300615 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:28.300568    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:11:28.317646 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:28.317595    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp" podStartSLOduration=1.317578247 podStartE2EDuration="1.317578247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:11:28.315846901 +0000 UTC m=+1573.127194275" watchObservedRunningTime="2026-07-27 22:11:28.317578247 +0000 UTC m=+1573.128925619"
Jul 27 22:11:28.336500 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:28.336453    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6" podStartSLOduration=1.925892478 podStartE2EDuration="2.336436909s" totalImagesPullingTime="410.544431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:11:28.334657181 +0000 UTC m=+1573.146004553" watchObservedRunningTime="2026-07-27 22:11:28.336436909 +0000 UTC m=+1573.147784280"
Jul 27 22:11:29.308427 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:29.308388    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b" event={"ID":"5d35f38f-85a2-4e0b-8969-7757ed583e8b","Type":"ContainerStarted","Data":"7e907a42ef65deba731c4dd2937cf3efa9acf48be898f8c52bc0b3c8f0879809"}
Jul 27 22:11:29.308961 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:29.308702    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"
Jul 27 22:11:29.329759 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:29.329712    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b" podStartSLOduration=2.3296942449999998 podStartE2EDuration="2.329694245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:11:29.327054388 +0000 UTC m=+1574.138401750" watchObservedRunningTime="2026-07-27 22:11:29.329694245 +0000 UTC m=+1574.141041618"
Jul 27 22:11:34.314207 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:34.314170    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:11:39.311425 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:39.311385    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"
Jul 27 22:11:40.315627 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:11:40.315598    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"
Jul 27 22:12:03.206862 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.206820    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"]
Jul 27 22:12:03.207601 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.207066    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6" podUID="8b619819-cc49-49ea-b448-a03abe8e2a0d" containerName="maas-api" containerID="cri-o://c07a6a9485aee0ff690f86fcf2bf8091e8d9dfd969b842871b81451b8b7bafdb" gracePeriod=30
Jul 27 22:12:03.278677 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.278642    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"]
Jul 27 22:12:03.278891 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.278864    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b" podUID="5d35f38f-85a2-4e0b-8969-7757ed583e8b" containerName="payload-processing" containerID="cri-o://7e907a42ef65deba731c4dd2937cf3efa9acf48be898f8c52bc0b3c8f0879809" gracePeriod=30
Jul 27 22:12:03.290926 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.290897    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"]
Jul 27 22:12:03.291136 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.291113    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp" podUID="cd81a934-def5-4a06-aff0-58595a0194f7" containerName="payload-pre-processing" containerID="cri-o://beef416bf8a9694d74df47a776348fade79ee44425ed171541c77b90c4242cc5" gracePeriod=30
Jul 27 22:12:03.484847 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.484542    2549 generic.go:356] "Generic (PLEG): container finished" podID="5d35f38f-85a2-4e0b-8969-7757ed583e8b" containerID="7e907a42ef65deba731c4dd2937cf3efa9acf48be898f8c52bc0b3c8f0879809" exitCode=0
Jul 27 22:12:03.484847 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.484604    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b" event={"ID":"5d35f38f-85a2-4e0b-8969-7757ed583e8b","Type":"ContainerDied","Data":"7e907a42ef65deba731c4dd2937cf3efa9acf48be898f8c52bc0b3c8f0879809"}
Jul 27 22:12:03.486960 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.486774    2549 generic.go:356] "Generic (PLEG): container finished" podID="cd81a934-def5-4a06-aff0-58595a0194f7" containerID="beef416bf8a9694d74df47a776348fade79ee44425ed171541c77b90c4242cc5" exitCode=0
Jul 27 22:12:03.486960 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.486882    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp" event={"ID":"cd81a934-def5-4a06-aff0-58595a0194f7","Type":"ContainerDied","Data":"beef416bf8a9694d74df47a776348fade79ee44425ed171541c77b90c4242cc5"}
Jul 27 22:12:03.488992 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.488969    2549 generic.go:356] "Generic (PLEG): container finished" podID="8b619819-cc49-49ea-b448-a03abe8e2a0d" containerID="c07a6a9485aee0ff690f86fcf2bf8091e8d9dfd969b842871b81451b8b7bafdb" exitCode=0
Jul 27 22:12:03.489116 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.489046    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6" event={"ID":"8b619819-cc49-49ea-b448-a03abe8e2a0d","Type":"ContainerDied","Data":"c07a6a9485aee0ff690f86fcf2bf8091e8d9dfd969b842871b81451b8b7bafdb"}
Jul 27 22:12:03.553007 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.552974    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"
Jul 27 22:12:03.573694 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.573663    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:12:03.610843 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.610815    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"
Jul 27 22:12:03.624633 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.624604    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6xkq\" (UniqueName: \"kubernetes.io/projected/cd81a934-def5-4a06-aff0-58595a0194f7-kube-api-access-l6xkq\") pod \"cd81a934-def5-4a06-aff0-58595a0194f7\" (UID: \"cd81a934-def5-4a06-aff0-58595a0194f7\") "
Jul 27 22:12:03.624772 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.624651    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd81a934-def5-4a06-aff0-58595a0194f7-plugins-config-volume\") pod \"cd81a934-def5-4a06-aff0-58595a0194f7\" (UID: \"cd81a934-def5-4a06-aff0-58595a0194f7\") "
Jul 27 22:12:03.625001 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.624981    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd81a934-def5-4a06-aff0-58595a0194f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cd81a934-def5-4a06-aff0-58595a0194f7" (UID: "cd81a934-def5-4a06-aff0-58595a0194f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:12:03.626874 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.626851    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd81a934-def5-4a06-aff0-58595a0194f7-kube-api-access-l6xkq" (OuterVolumeSpecName: "kube-api-access-l6xkq") pod "cd81a934-def5-4a06-aff0-58595a0194f7" (UID: "cd81a934-def5-4a06-aff0-58595a0194f7"). InnerVolumeSpecName "kube-api-access-l6xkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:12:03.725965 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.725893    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvmhw\" (UniqueName: \"kubernetes.io/projected/5d35f38f-85a2-4e0b-8969-7757ed583e8b-kube-api-access-fvmhw\") pod \"5d35f38f-85a2-4e0b-8969-7757ed583e8b\" (UID: \"5d35f38f-85a2-4e0b-8969-7757ed583e8b\") "
Jul 27 22:12:03.725965 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.725941    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b619819-cc49-49ea-b448-a03abe8e2a0d-maas-api-tls\") pod \"8b619819-cc49-49ea-b448-a03abe8e2a0d\" (UID: \"8b619819-cc49-49ea-b448-a03abe8e2a0d\") "
Jul 27 22:12:03.725965 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.725965    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d35f38f-85a2-4e0b-8969-7757ed583e8b-plugins-config-volume\") pod \"5d35f38f-85a2-4e0b-8969-7757ed583e8b\" (UID: \"5d35f38f-85a2-4e0b-8969-7757ed583e8b\") "
Jul 27 22:12:03.726207 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.725986    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rktjj\" (UniqueName: \"kubernetes.io/projected/8b619819-cc49-49ea-b448-a03abe8e2a0d-kube-api-access-rktjj\") pod \"8b619819-cc49-49ea-b448-a03abe8e2a0d\" (UID: \"8b619819-cc49-49ea-b448-a03abe8e2a0d\") "
Jul 27 22:12:03.726207 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.726009    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b619819-cc49-49ea-b448-a03abe8e2a0d-openshift-service-ca\") pod \"8b619819-cc49-49ea-b448-a03abe8e2a0d\" (UID: \"8b619819-cc49-49ea-b448-a03abe8e2a0d\") "
Jul 27 22:12:03.726340 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.726255    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6xkq\" (UniqueName: \"kubernetes.io/projected/cd81a934-def5-4a06-aff0-58595a0194f7-kube-api-access-l6xkq\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:12:03.726340 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.726275    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd81a934-def5-4a06-aff0-58595a0194f7-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:12:03.726440 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.726396    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d35f38f-85a2-4e0b-8969-7757ed583e8b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d35f38f-85a2-4e0b-8969-7757ed583e8b" (UID: "5d35f38f-85a2-4e0b-8969-7757ed583e8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:12:03.726499 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.726428    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b619819-cc49-49ea-b448-a03abe8e2a0d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b619819-cc49-49ea-b448-a03abe8e2a0d" (UID: "8b619819-cc49-49ea-b448-a03abe8e2a0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:12:03.728184 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.728158    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d35f38f-85a2-4e0b-8969-7757ed583e8b-kube-api-access-fvmhw" (OuterVolumeSpecName: "kube-api-access-fvmhw") pod "5d35f38f-85a2-4e0b-8969-7757ed583e8b" (UID: "5d35f38f-85a2-4e0b-8969-7757ed583e8b"). InnerVolumeSpecName "kube-api-access-fvmhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:12:03.728345 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.728324    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b619819-cc49-49ea-b448-a03abe8e2a0d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b619819-cc49-49ea-b448-a03abe8e2a0d" (UID: "8b619819-cc49-49ea-b448-a03abe8e2a0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:12:03.728345 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.728331    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b619819-cc49-49ea-b448-a03abe8e2a0d-kube-api-access-rktjj" (OuterVolumeSpecName: "kube-api-access-rktjj") pod "8b619819-cc49-49ea-b448-a03abe8e2a0d" (UID: "8b619819-cc49-49ea-b448-a03abe8e2a0d"). InnerVolumeSpecName "kube-api-access-rktjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:12:03.826993 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.826940    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b619819-cc49-49ea-b448-a03abe8e2a0d-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:12:03.826993 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.826981    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvmhw\" (UniqueName: \"kubernetes.io/projected/5d35f38f-85a2-4e0b-8969-7757ed583e8b-kube-api-access-fvmhw\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:12:03.827206 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.827005    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b619819-cc49-49ea-b448-a03abe8e2a0d-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:12:03.827206 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.827018    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d35f38f-85a2-4e0b-8969-7757ed583e8b-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:12:03.827206 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:03.827030    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rktjj\" (UniqueName: \"kubernetes.io/projected/8b619819-cc49-49ea-b448-a03abe8e2a0d-kube-api-access-rktjj\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:12:04.497780 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.497693    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"
Jul 27 22:12:04.497780 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.497764    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6" event={"ID":"8b619819-cc49-49ea-b448-a03abe8e2a0d","Type":"ContainerDied","Data":"b205d2ce765d3851fd644372088d08e316d606fa4ef840eee693b3f6d66b5609"}
Jul 27 22:12:04.498286 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.497821    2549 scope.go:117] "RemoveContainer" containerID="c07a6a9485aee0ff690f86fcf2bf8091e8d9dfd969b842871b81451b8b7bafdb"
Jul 27 22:12:04.499528 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.499317    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"
Jul 27 22:12:04.499528 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.499331    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b" event={"ID":"5d35f38f-85a2-4e0b-8969-7757ed583e8b","Type":"ContainerDied","Data":"b6730cae8b2f6756a85b4cdf86c6d262f8ffb6c59cc0f5510d8a92766aa8fa19"}
Jul 27 22:12:04.500863 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.500838    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp" event={"ID":"cd81a934-def5-4a06-aff0-58595a0194f7","Type":"ContainerDied","Data":"bda2fb1da88d8b5b00937f8471fd6b4971828fe08301d1e48ca8cd35efd81cd0"}
Jul 27 22:12:04.500969 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.500865    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"
Jul 27 22:12:04.508028 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.507923    2549 scope.go:117] "RemoveContainer" containerID="7e907a42ef65deba731c4dd2937cf3efa9acf48be898f8c52bc0b3c8f0879809"
Jul 27 22:12:04.516298 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.516284    2549 scope.go:117] "RemoveContainer" containerID="beef416bf8a9694d74df47a776348fade79ee44425ed171541c77b90c4242cc5"
Jul 27 22:12:04.525553 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.525530    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"]
Jul 27 22:12:04.530493 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.530469    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a66e79c4-fd67458f4-j4w9b"]
Jul 27 22:12:04.543568 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.543534    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"]
Jul 27 22:12:04.546414 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.546389    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a66e79c4-5569c7f464-khjfp"]
Jul 27 22:12:04.560561 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.560536    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"]
Jul 27 22:12:04.563753 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:04.563725    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a66e79c4-566988998-c9td6"]
Jul 27 22:12:05.699992 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:05.699954    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d35f38f-85a2-4e0b-8969-7757ed583e8b" path="/var/lib/kubelet/pods/5d35f38f-85a2-4e0b-8969-7757ed583e8b/volumes"
Jul 27 22:12:05.700356 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:05.700347    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b619819-cc49-49ea-b448-a03abe8e2a0d" path="/var/lib/kubelet/pods/8b619819-cc49-49ea-b448-a03abe8e2a0d/volumes"
Jul 27 22:12:05.700652 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:05.700639    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd81a934-def5-4a06-aff0-58595a0194f7" path="/var/lib/kubelet/pods/cd81a934-def5-4a06-aff0-58595a0194f7/volumes"
Jul 27 22:12:12.353861 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.353825    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"]
Jul 27 22:12:12.354344 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.354315    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd81a934-def5-4a06-aff0-58595a0194f7" containerName="payload-pre-processing"
Jul 27 22:12:12.354344 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.354338    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd81a934-def5-4a06-aff0-58595a0194f7" containerName="payload-pre-processing"
Jul 27 22:12:12.354470 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.354354    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d35f38f-85a2-4e0b-8969-7757ed583e8b" containerName="payload-processing"
Jul 27 22:12:12.354470 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.354368    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d35f38f-85a2-4e0b-8969-7757ed583e8b" containerName="payload-processing"
Jul 27 22:12:12.354470 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.354398    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b619819-cc49-49ea-b448-a03abe8e2a0d" containerName="maas-api"
Jul 27 22:12:12.354470 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.354409    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b619819-cc49-49ea-b448-a03abe8e2a0d" containerName="maas-api"
Jul 27 22:12:12.354640 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.354513    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b619819-cc49-49ea-b448-a03abe8e2a0d" containerName="maas-api"
Jul 27 22:12:12.354640 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.354531    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd81a934-def5-4a06-aff0-58595a0194f7" containerName="payload-pre-processing"
Jul 27 22:12:12.354640 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.354541    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d35f38f-85a2-4e0b-8969-7757ed583e8b" containerName="payload-processing"
Jul 27 22:12:12.358792 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.358773    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:12.361772 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.361749    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4d2830c6-serving-cert\""
Jul 27 22:12:12.372153 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.372127    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"]
Jul 27 22:12:12.502544 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.502512    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0258db-795e-4449-8b68-b14fde05e45a-maas-api-tls\") pod \"maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5\" (UID: \"de0258db-795e-4449-8b68-b14fde05e45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:12.502544 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.502545    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0258db-795e-4449-8b68-b14fde05e45a-openshift-service-ca\") pod \"maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5\" (UID: \"de0258db-795e-4449-8b68-b14fde05e45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:12.502752 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.502569    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75jjm\" (UniqueName: \"kubernetes.io/projected/de0258db-795e-4449-8b68-b14fde05e45a-kube-api-access-75jjm\") pod \"maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5\" (UID: \"de0258db-795e-4449-8b68-b14fde05e45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:12.603525 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.603492    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0258db-795e-4449-8b68-b14fde05e45a-maas-api-tls\") pod \"maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5\" (UID: \"de0258db-795e-4449-8b68-b14fde05e45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:12.603525 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.603525    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0258db-795e-4449-8b68-b14fde05e45a-openshift-service-ca\") pod \"maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5\" (UID: \"de0258db-795e-4449-8b68-b14fde05e45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:12.603860 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.603548    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75jjm\" (UniqueName: \"kubernetes.io/projected/de0258db-795e-4449-8b68-b14fde05e45a-kube-api-access-75jjm\") pod \"maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5\" (UID: \"de0258db-795e-4449-8b68-b14fde05e45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:12.603860 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:12:12.603659    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-serving-cert: secret "maas-api-e2e-migrate-4d2830c6-serving-cert" not found
Jul 27 22:12:12.603860 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:12:12.603739    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0258db-795e-4449-8b68-b14fde05e45a-maas-api-tls podName:de0258db-795e-4449-8b68-b14fde05e45a nodeName:}" failed. No retries permitted until 2026-07-27 22:12:13.103721181 +0000 UTC m=+1617.915068531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de0258db-795e-4449-8b68-b14fde05e45a-maas-api-tls") pod "maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5" (UID: "de0258db-795e-4449-8b68-b14fde05e45a") : secret "maas-api-e2e-migrate-4d2830c6-serving-cert" not found
Jul 27 22:12:12.604296 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.604211    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0258db-795e-4449-8b68-b14fde05e45a-openshift-service-ca\") pod \"maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5\" (UID: \"de0258db-795e-4449-8b68-b14fde05e45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:12.613139 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.613115    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75jjm\" (UniqueName: \"kubernetes.io/projected/de0258db-795e-4449-8b68-b14fde05e45a-kube-api-access-75jjm\") pod \"maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5\" (UID: \"de0258db-795e-4449-8b68-b14fde05e45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:12.971427 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.971397    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"]
Jul 27 22:12:12.974806 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.974787    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"
Jul 27 22:12:12.977778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.977746    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4d2830c6\""
Jul 27 22:12:12.977778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.977746    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4d2830c6-dockercfg-zgf7c\""
Jul 27 22:12:12.986816 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:12.986789    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"]
Jul 27 22:12:13.002307 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.002277    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"]
Jul 27 22:12:13.006452 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.006430    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"
Jul 27 22:12:13.015571 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.015535    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"]
Jul 27 22:12:13.107656 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.107622    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2frw\" (UniqueName: \"kubernetes.io/projected/9472e5a3-063a-4d26-9722-fb34edadff5b-kube-api-access-r2frw\") pod \"payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv\" (UID: \"9472e5a3-063a-4d26-9722-fb34edadff5b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"
Jul 27 22:12:13.107843 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.107669    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9472e5a3-063a-4d26-9722-fb34edadff5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv\" (UID: \"9472e5a3-063a-4d26-9722-fb34edadff5b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"
Jul 27 22:12:13.107843 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.107712    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da84478-17b3-4036-ba17-4d97fdbe1ae6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5\" (UID: \"1da84478-17b3-4036-ba17-4d97fdbe1ae6\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"
Jul 27 22:12:13.107843 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.107752    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0258db-795e-4449-8b68-b14fde05e45a-maas-api-tls\") pod \"maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5\" (UID: \"de0258db-795e-4449-8b68-b14fde05e45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:13.107843 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.107781    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbd9f\" (UniqueName: \"kubernetes.io/projected/1da84478-17b3-4036-ba17-4d97fdbe1ae6-kube-api-access-pbd9f\") pod \"payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5\" (UID: \"1da84478-17b3-4036-ba17-4d97fdbe1ae6\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"
Jul 27 22:12:13.110425 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.110398    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0258db-795e-4449-8b68-b14fde05e45a-maas-api-tls\") pod \"maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5\" (UID: \"de0258db-795e-4449-8b68-b14fde05e45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:13.208834 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.208800    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da84478-17b3-4036-ba17-4d97fdbe1ae6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5\" (UID: \"1da84478-17b3-4036-ba17-4d97fdbe1ae6\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"
Jul 27 22:12:13.209028 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.208861    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbd9f\" (UniqueName: \"kubernetes.io/projected/1da84478-17b3-4036-ba17-4d97fdbe1ae6-kube-api-access-pbd9f\") pod \"payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5\" (UID: \"1da84478-17b3-4036-ba17-4d97fdbe1ae6\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"
Jul 27 22:12:13.209028 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.208995    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2frw\" (UniqueName: \"kubernetes.io/projected/9472e5a3-063a-4d26-9722-fb34edadff5b-kube-api-access-r2frw\") pod \"payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv\" (UID: \"9472e5a3-063a-4d26-9722-fb34edadff5b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"
Jul 27 22:12:13.209155 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.209043    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9472e5a3-063a-4d26-9722-fb34edadff5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv\" (UID: \"9472e5a3-063a-4d26-9722-fb34edadff5b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"
Jul 27 22:12:13.209668 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.209644    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da84478-17b3-4036-ba17-4d97fdbe1ae6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5\" (UID: \"1da84478-17b3-4036-ba17-4d97fdbe1ae6\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"
Jul 27 22:12:13.209769 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.209690    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9472e5a3-063a-4d26-9722-fb34edadff5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv\" (UID: \"9472e5a3-063a-4d26-9722-fb34edadff5b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"
Jul 27 22:12:13.222687 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.222616    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2frw\" (UniqueName: \"kubernetes.io/projected/9472e5a3-063a-4d26-9722-fb34edadff5b-kube-api-access-r2frw\") pod \"payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv\" (UID: \"9472e5a3-063a-4d26-9722-fb34edadff5b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"
Jul 27 22:12:13.232442 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.232420    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbd9f\" (UniqueName: \"kubernetes.io/projected/1da84478-17b3-4036-ba17-4d97fdbe1ae6-kube-api-access-pbd9f\") pod \"payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5\" (UID: \"1da84478-17b3-4036-ba17-4d97fdbe1ae6\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"
Jul 27 22:12:13.270386 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.270351    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:13.286448 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.286421    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"
Jul 27 22:12:13.321009 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.320793    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"
Jul 27 22:12:13.447265 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.447203    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"]
Jul 27 22:12:13.449677 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:12:13.449648    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0258db_795e_4449_8b68_b14fde05e45a.slice/crio-49a401bd94429dc6794531c2c933c155b49c87c20886db56a4cdd49c4a0c13a0 WatchSource:0}: Error finding container 49a401bd94429dc6794531c2c933c155b49c87c20886db56a4cdd49c4a0c13a0: Status 404 returned error can't find the container with id 49a401bd94429dc6794531c2c933c155b49c87c20886db56a4cdd49c4a0c13a0
Jul 27 22:12:13.508966 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.508934    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"]
Jul 27 22:12:13.510403 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:12:13.510362    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da84478_17b3_4036_ba17_4d97fdbe1ae6.slice/crio-b5d6d7da258f09e405a3297ab1337386bca164077ae1218e31d0b69eaa71a8f6 WatchSource:0}: Error finding container b5d6d7da258f09e405a3297ab1337386bca164077ae1218e31d0b69eaa71a8f6: Status 404 returned error can't find the container with id b5d6d7da258f09e405a3297ab1337386bca164077ae1218e31d0b69eaa71a8f6
Jul 27 22:12:13.554931 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.554898    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5" event={"ID":"1da84478-17b3-4036-ba17-4d97fdbe1ae6","Type":"ContainerStarted","Data":"b5d6d7da258f09e405a3297ab1337386bca164077ae1218e31d0b69eaa71a8f6"}
Jul 27 22:12:13.556284 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.556261    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5" event={"ID":"de0258db-795e-4449-8b68-b14fde05e45a","Type":"ContainerStarted","Data":"49a401bd94429dc6794531c2c933c155b49c87c20886db56a4cdd49c4a0c13a0"}
Jul 27 22:12:13.674265 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:13.674088    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"]
Jul 27 22:12:13.676967 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:12:13.676943    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9472e5a3_063a_4d26_9722_fb34edadff5b.slice/crio-38cadbefdeb838eb025ac1df218c310d5432ada9a140b51987ae6a0e716b2eaa WatchSource:0}: Error finding container 38cadbefdeb838eb025ac1df218c310d5432ada9a140b51987ae6a0e716b2eaa: Status 404 returned error can't find the container with id 38cadbefdeb838eb025ac1df218c310d5432ada9a140b51987ae6a0e716b2eaa
Jul 27 22:12:14.563394 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:14.563342    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5" event={"ID":"de0258db-795e-4449-8b68-b14fde05e45a","Type":"ContainerStarted","Data":"4dabc18caff35cc120d2a517aeec52a376e0d5002b51a59698310a3367de2925"}
Jul 27 22:12:14.563912 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:14.563419    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:14.564903 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:14.564872    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv" event={"ID":"9472e5a3-063a-4d26-9722-fb34edadff5b","Type":"ContainerStarted","Data":"ee89b0625fc851479377a1ba8d16284051fa89cc1e44944c971d8c7026fe0441"}
Jul 27 22:12:14.565027 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:14.564918    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv" event={"ID":"9472e5a3-063a-4d26-9722-fb34edadff5b","Type":"ContainerStarted","Data":"38cadbefdeb838eb025ac1df218c310d5432ada9a140b51987ae6a0e716b2eaa"}
Jul 27 22:12:14.565027 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:14.564944    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"
Jul 27 22:12:14.566282 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:14.566258    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5" event={"ID":"1da84478-17b3-4036-ba17-4d97fdbe1ae6","Type":"ContainerStarted","Data":"26226ea29533176cc0f2eae5742cf788f015db906adc10499dc9ca9a85466463"}
Jul 27 22:12:14.566375 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:14.566303    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"
Jul 27 22:12:14.583807 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:14.583757    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5" podStartSLOduration=2.2297424120000002 podStartE2EDuration="2.583743721s" totalImagesPullingTime="354.001309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:12:14.583397135 +0000 UTC m=+1619.394744506" watchObservedRunningTime="2026-07-27 22:12:14.583743721 +0000 UTC m=+1619.395091093"
Jul 27 22:12:14.619231 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:14.619184    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv" podStartSLOduration=2.6191698580000002 podStartE2EDuration="2.619169858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:12:14.602247733 +0000 UTC m=+1619.413595103" watchObservedRunningTime="2026-07-27 22:12:14.619169858 +0000 UTC m=+1619.430517266"
Jul 27 22:12:14.620514 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:14.620478    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5" podStartSLOduration=2.620463273 podStartE2EDuration="2.620463273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:12:14.619040475 +0000 UTC m=+1619.430387850" watchObservedRunningTime="2026-07-27 22:12:14.620463273 +0000 UTC m=+1619.431810646"
Jul 27 22:12:20.578476 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:20.578448    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:25.574731 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:25.574693    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"
Jul 27 22:12:25.575110 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:25.574894    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"
Jul 27 22:12:47.633458 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:47.633421    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"]
Jul 27 22:12:47.633993 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:47.633700    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5" podUID="de0258db-795e-4449-8b68-b14fde05e45a" containerName="maas-api" containerID="cri-o://4dabc18caff35cc120d2a517aeec52a376e0d5002b51a59698310a3367de2925" gracePeriod=30
Jul 27 22:12:47.705071 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:47.705034    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"]
Jul 27 22:12:47.705283 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:47.705113    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"]
Jul 27 22:12:47.705394 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:47.705365    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv" podUID="9472e5a3-063a-4d26-9722-fb34edadff5b" containerName="payload-pre-processing" containerID="cri-o://ee89b0625fc851479377a1ba8d16284051fa89cc1e44944c971d8c7026fe0441" gracePeriod=30
Jul 27 22:12:47.705605 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:47.705567    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5" podUID="1da84478-17b3-4036-ba17-4d97fdbe1ae6" containerName="payload-processing" containerID="cri-o://26226ea29533176cc0f2eae5742cf788f015db906adc10499dc9ca9a85466463" gracePeriod=30
Jul 27 22:12:47.968849 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:47.968816    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:48.008326 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.008283    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"
Jul 27 22:12:48.019456 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.019429    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"
Jul 27 22:12:48.021321 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.021296    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75jjm\" (UniqueName: \"kubernetes.io/projected/de0258db-795e-4449-8b68-b14fde05e45a-kube-api-access-75jjm\") pod \"de0258db-795e-4449-8b68-b14fde05e45a\" (UID: \"de0258db-795e-4449-8b68-b14fde05e45a\") "
Jul 27 22:12:48.021483 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.021333    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0258db-795e-4449-8b68-b14fde05e45a-openshift-service-ca\") pod \"de0258db-795e-4449-8b68-b14fde05e45a\" (UID: \"de0258db-795e-4449-8b68-b14fde05e45a\") "
Jul 27 22:12:48.021483 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.021446    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0258db-795e-4449-8b68-b14fde05e45a-maas-api-tls\") pod \"de0258db-795e-4449-8b68-b14fde05e45a\" (UID: \"de0258db-795e-4449-8b68-b14fde05e45a\") "
Jul 27 22:12:48.021667 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.021642    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0258db-795e-4449-8b68-b14fde05e45a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "de0258db-795e-4449-8b68-b14fde05e45a" (UID: "de0258db-795e-4449-8b68-b14fde05e45a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:12:48.021777 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.021760    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0258db-795e-4449-8b68-b14fde05e45a-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:12:48.023544 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.023516    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0258db-795e-4449-8b68-b14fde05e45a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "de0258db-795e-4449-8b68-b14fde05e45a" (UID: "de0258db-795e-4449-8b68-b14fde05e45a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:12:48.023779 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.023749    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0258db-795e-4449-8b68-b14fde05e45a-kube-api-access-75jjm" (OuterVolumeSpecName: "kube-api-access-75jjm") pod "de0258db-795e-4449-8b68-b14fde05e45a" (UID: "de0258db-795e-4449-8b68-b14fde05e45a"). InnerVolumeSpecName "kube-api-access-75jjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:12:48.122406 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.122374    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da84478-17b3-4036-ba17-4d97fdbe1ae6-plugins-config-volume\") pod \"1da84478-17b3-4036-ba17-4d97fdbe1ae6\" (UID: \"1da84478-17b3-4036-ba17-4d97fdbe1ae6\") "
Jul 27 22:12:48.122588 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.122444    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2frw\" (UniqueName: \"kubernetes.io/projected/9472e5a3-063a-4d26-9722-fb34edadff5b-kube-api-access-r2frw\") pod \"9472e5a3-063a-4d26-9722-fb34edadff5b\" (UID: \"9472e5a3-063a-4d26-9722-fb34edadff5b\") "
Jul 27 22:12:48.122588 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.122483    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9472e5a3-063a-4d26-9722-fb34edadff5b-plugins-config-volume\") pod \"9472e5a3-063a-4d26-9722-fb34edadff5b\" (UID: \"9472e5a3-063a-4d26-9722-fb34edadff5b\") "
Jul 27 22:12:48.122588 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.122534    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbd9f\" (UniqueName: \"kubernetes.io/projected/1da84478-17b3-4036-ba17-4d97fdbe1ae6-kube-api-access-pbd9f\") pod \"1da84478-17b3-4036-ba17-4d97fdbe1ae6\" (UID: \"1da84478-17b3-4036-ba17-4d97fdbe1ae6\") "
Jul 27 22:12:48.122835 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.122805    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0258db-795e-4449-8b68-b14fde05e45a-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:12:48.122955 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.122846    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75jjm\" (UniqueName: \"kubernetes.io/projected/de0258db-795e-4449-8b68-b14fde05e45a-kube-api-access-75jjm\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:12:48.122955 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.122808    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da84478-17b3-4036-ba17-4d97fdbe1ae6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1da84478-17b3-4036-ba17-4d97fdbe1ae6" (UID: "1da84478-17b3-4036-ba17-4d97fdbe1ae6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:12:48.122955 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.122850    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9472e5a3-063a-4d26-9722-fb34edadff5b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9472e5a3-063a-4d26-9722-fb34edadff5b" (UID: "9472e5a3-063a-4d26-9722-fb34edadff5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:12:48.124654 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.124633    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da84478-17b3-4036-ba17-4d97fdbe1ae6-kube-api-access-pbd9f" (OuterVolumeSpecName: "kube-api-access-pbd9f") pod "1da84478-17b3-4036-ba17-4d97fdbe1ae6" (UID: "1da84478-17b3-4036-ba17-4d97fdbe1ae6"). InnerVolumeSpecName "kube-api-access-pbd9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:12:48.124761 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.124743    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9472e5a3-063a-4d26-9722-fb34edadff5b-kube-api-access-r2frw" (OuterVolumeSpecName: "kube-api-access-r2frw") pod "9472e5a3-063a-4d26-9722-fb34edadff5b" (UID: "9472e5a3-063a-4d26-9722-fb34edadff5b"). InnerVolumeSpecName "kube-api-access-r2frw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:12:48.223961 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.223860    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9472e5a3-063a-4d26-9722-fb34edadff5b-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:12:48.223961 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.223898    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbd9f\" (UniqueName: \"kubernetes.io/projected/1da84478-17b3-4036-ba17-4d97fdbe1ae6-kube-api-access-pbd9f\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:12:48.223961 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.223910    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da84478-17b3-4036-ba17-4d97fdbe1ae6-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:12:48.223961 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.223922    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2frw\" (UniqueName: \"kubernetes.io/projected/9472e5a3-063a-4d26-9722-fb34edadff5b-kube-api-access-r2frw\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:12:48.746719 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.746685    2549 generic.go:356] "Generic (PLEG): container finished" podID="de0258db-795e-4449-8b68-b14fde05e45a" containerID="4dabc18caff35cc120d2a517aeec52a376e0d5002b51a59698310a3367de2925" exitCode=0
Jul 27 22:12:48.747168 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.746731    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"
Jul 27 22:12:48.747168 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.746757    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5" event={"ID":"de0258db-795e-4449-8b68-b14fde05e45a","Type":"ContainerDied","Data":"4dabc18caff35cc120d2a517aeec52a376e0d5002b51a59698310a3367de2925"}
Jul 27 22:12:48.747168 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.746812    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5" event={"ID":"de0258db-795e-4449-8b68-b14fde05e45a","Type":"ContainerDied","Data":"49a401bd94429dc6794531c2c933c155b49c87c20886db56a4cdd49c4a0c13a0"}
Jul 27 22:12:48.747168 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.746843    2549 scope.go:117] "RemoveContainer" containerID="4dabc18caff35cc120d2a517aeec52a376e0d5002b51a59698310a3367de2925"
Jul 27 22:12:48.748265 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.748241    2549 generic.go:356] "Generic (PLEG): container finished" podID="9472e5a3-063a-4d26-9722-fb34edadff5b" containerID="ee89b0625fc851479377a1ba8d16284051fa89cc1e44944c971d8c7026fe0441" exitCode=0
Jul 27 22:12:48.748341 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.748291    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"
Jul 27 22:12:48.748341 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.748311    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv" event={"ID":"9472e5a3-063a-4d26-9722-fb34edadff5b","Type":"ContainerDied","Data":"ee89b0625fc851479377a1ba8d16284051fa89cc1e44944c971d8c7026fe0441"}
Jul 27 22:12:48.748446 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.748364    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv" event={"ID":"9472e5a3-063a-4d26-9722-fb34edadff5b","Type":"ContainerDied","Data":"38cadbefdeb838eb025ac1df218c310d5432ada9a140b51987ae6a0e716b2eaa"}
Jul 27 22:12:48.749732 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.749711    2549 generic.go:356] "Generic (PLEG): container finished" podID="1da84478-17b3-4036-ba17-4d97fdbe1ae6" containerID="26226ea29533176cc0f2eae5742cf788f015db906adc10499dc9ca9a85466463" exitCode=0
Jul 27 22:12:48.749856 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.749763    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"
Jul 27 22:12:48.749856 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.749781    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5" event={"ID":"1da84478-17b3-4036-ba17-4d97fdbe1ae6","Type":"ContainerDied","Data":"26226ea29533176cc0f2eae5742cf788f015db906adc10499dc9ca9a85466463"}
Jul 27 22:12:48.749856 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.749820    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5" event={"ID":"1da84478-17b3-4036-ba17-4d97fdbe1ae6","Type":"ContainerDied","Data":"b5d6d7da258f09e405a3297ab1337386bca164077ae1218e31d0b69eaa71a8f6"}
Jul 27 22:12:48.758731 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.758713    2549 scope.go:117] "RemoveContainer" containerID="4dabc18caff35cc120d2a517aeec52a376e0d5002b51a59698310a3367de2925"
Jul 27 22:12:48.759054 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:12:48.759020    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dabc18caff35cc120d2a517aeec52a376e0d5002b51a59698310a3367de2925\": container with ID starting with 4dabc18caff35cc120d2a517aeec52a376e0d5002b51a59698310a3367de2925 not found: ID does not exist" containerID="4dabc18caff35cc120d2a517aeec52a376e0d5002b51a59698310a3367de2925"
Jul 27 22:12:48.759164 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.759055    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dabc18caff35cc120d2a517aeec52a376e0d5002b51a59698310a3367de2925"} err="failed to get container status \"4dabc18caff35cc120d2a517aeec52a376e0d5002b51a59698310a3367de2925\": rpc error: code = NotFound desc = could not find container \"4dabc18caff35cc120d2a517aeec52a376e0d5002b51a59698310a3367de2925\": container with ID starting with 4dabc18caff35cc120d2a517aeec52a376e0d5002b51a59698310a3367de2925 not found: ID does not exist"
Jul 27 22:12:48.759164 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.759080    2549 scope.go:117] "RemoveContainer" containerID="ee89b0625fc851479377a1ba8d16284051fa89cc1e44944c971d8c7026fe0441"
Jul 27 22:12:48.768863 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.768827    2549 scope.go:117] "RemoveContainer" containerID="ee89b0625fc851479377a1ba8d16284051fa89cc1e44944c971d8c7026fe0441"
Jul 27 22:12:48.769238 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:12:48.769194    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee89b0625fc851479377a1ba8d16284051fa89cc1e44944c971d8c7026fe0441\": container with ID starting with ee89b0625fc851479377a1ba8d16284051fa89cc1e44944c971d8c7026fe0441 not found: ID does not exist" containerID="ee89b0625fc851479377a1ba8d16284051fa89cc1e44944c971d8c7026fe0441"
Jul 27 22:12:48.769339 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.769252    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee89b0625fc851479377a1ba8d16284051fa89cc1e44944c971d8c7026fe0441"} err="failed to get container status \"ee89b0625fc851479377a1ba8d16284051fa89cc1e44944c971d8c7026fe0441\": rpc error: code = NotFound desc = could not find container \"ee89b0625fc851479377a1ba8d16284051fa89cc1e44944c971d8c7026fe0441\": container with ID starting with ee89b0625fc851479377a1ba8d16284051fa89cc1e44944c971d8c7026fe0441 not found: ID does not exist"
Jul 27 22:12:48.769339 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.769282    2549 scope.go:117] "RemoveContainer" containerID="26226ea29533176cc0f2eae5742cf788f015db906adc10499dc9ca9a85466463"
Jul 27 22:12:48.771746 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.771722    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"]
Jul 27 22:12:48.775505 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.775483    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d2830c6-7656c7d874-68ts5"]
Jul 27 22:12:48.778245 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.778208    2549 scope.go:117] "RemoveContainer" containerID="26226ea29533176cc0f2eae5742cf788f015db906adc10499dc9ca9a85466463"
Jul 27 22:12:48.778513 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:12:48.778496    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26226ea29533176cc0f2eae5742cf788f015db906adc10499dc9ca9a85466463\": container with ID starting with 26226ea29533176cc0f2eae5742cf788f015db906adc10499dc9ca9a85466463 not found: ID does not exist" containerID="26226ea29533176cc0f2eae5742cf788f015db906adc10499dc9ca9a85466463"
Jul 27 22:12:48.778565 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.778518    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26226ea29533176cc0f2eae5742cf788f015db906adc10499dc9ca9a85466463"} err="failed to get container status \"26226ea29533176cc0f2eae5742cf788f015db906adc10499dc9ca9a85466463\": rpc error: code = NotFound desc = could not find container \"26226ea29533176cc0f2eae5742cf788f015db906adc10499dc9ca9a85466463\": container with ID starting with 26226ea29533176cc0f2eae5742cf788f015db906adc10499dc9ca9a85466463 not found: ID does not exist"
Jul 27 22:12:48.787483 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.787463    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"]
Jul 27 22:12:48.791344 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.791325    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d2830c6-6c78ff6cfb-vdvmv"]
Jul 27 22:12:48.802496 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.802463    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"]
Jul 27 22:12:48.807370 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:48.807350    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d2830c6-546988c685-7d9l5"]
Jul 27 22:12:49.699977 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:49.699945    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1da84478-17b3-4036-ba17-4d97fdbe1ae6" path="/var/lib/kubelet/pods/1da84478-17b3-4036-ba17-4d97fdbe1ae6/volumes"
Jul 27 22:12:49.700339 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:49.700325    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9472e5a3-063a-4d26-9722-fb34edadff5b" path="/var/lib/kubelet/pods/9472e5a3-063a-4d26-9722-fb34edadff5b/volumes"
Jul 27 22:12:49.700674 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:49.700662    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de0258db-795e-4449-8b68-b14fde05e45a" path="/var/lib/kubelet/pods/de0258db-795e-4449-8b68-b14fde05e45a/volumes"
Jul 27 22:12:55.476206 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.476170    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"]
Jul 27 22:12:55.476802 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.476687    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da84478-17b3-4036-ba17-4d97fdbe1ae6" containerName="payload-processing"
Jul 27 22:12:55.476802 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.476712    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da84478-17b3-4036-ba17-4d97fdbe1ae6" containerName="payload-processing"
Jul 27 22:12:55.476802 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.476731    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9472e5a3-063a-4d26-9722-fb34edadff5b" containerName="payload-pre-processing"
Jul 27 22:12:55.476802 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.476740    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="9472e5a3-063a-4d26-9722-fb34edadff5b" containerName="payload-pre-processing"
Jul 27 22:12:55.476802 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.476750    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0258db-795e-4449-8b68-b14fde05e45a" containerName="maas-api"
Jul 27 22:12:55.476802 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.476760    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0258db-795e-4449-8b68-b14fde05e45a" containerName="maas-api"
Jul 27 22:12:55.477193 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.476881    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="9472e5a3-063a-4d26-9722-fb34edadff5b" containerName="payload-pre-processing"
Jul 27 22:12:55.477193 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.476897    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da84478-17b3-4036-ba17-4d97fdbe1ae6" containerName="payload-processing"
Jul 27 22:12:55.477193 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.476911    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="de0258db-795e-4449-8b68-b14fde05e45a" containerName="maas-api"
Jul 27 22:12:55.481993 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.481969    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:12:55.486739 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.486713    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d4157b7d-serving-cert\""
Jul 27 22:12:55.498002 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.497910    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"]
Jul 27 22:12:55.591469 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.591426    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae968e-dd25-4c4c-8384-0b49e7137283-openshift-service-ca\") pod \"maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt\" (UID: \"aaae968e-dd25-4c4c-8384-0b49e7137283\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:12:55.591633 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.591546    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaae968e-dd25-4c4c-8384-0b49e7137283-maas-api-tls\") pod \"maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt\" (UID: \"aaae968e-dd25-4c4c-8384-0b49e7137283\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:12:55.591633 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.591573    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dms6h\" (UniqueName: \"kubernetes.io/projected/aaae968e-dd25-4c4c-8384-0b49e7137283-kube-api-access-dms6h\") pod \"maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt\" (UID: \"aaae968e-dd25-4c4c-8384-0b49e7137283\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:12:55.692400 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.692354    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaae968e-dd25-4c4c-8384-0b49e7137283-maas-api-tls\") pod \"maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt\" (UID: \"aaae968e-dd25-4c4c-8384-0b49e7137283\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:12:55.692400 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.692402    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dms6h\" (UniqueName: \"kubernetes.io/projected/aaae968e-dd25-4c4c-8384-0b49e7137283-kube-api-access-dms6h\") pod \"maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt\" (UID: \"aaae968e-dd25-4c4c-8384-0b49e7137283\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:12:55.692664 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.692478    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae968e-dd25-4c4c-8384-0b49e7137283-openshift-service-ca\") pod \"maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt\" (UID: \"aaae968e-dd25-4c4c-8384-0b49e7137283\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:12:55.692664 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:12:55.692513    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-serving-cert: secret "maas-api-e2e-ait-d4157b7d-serving-cert" not found
Jul 27 22:12:55.692664 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:12:55.692577    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaae968e-dd25-4c4c-8384-0b49e7137283-maas-api-tls podName:aaae968e-dd25-4c4c-8384-0b49e7137283 nodeName:}" failed. No retries permitted until 2026-07-27 22:12:56.192561466 +0000 UTC m=+1661.003908817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aaae968e-dd25-4c4c-8384-0b49e7137283-maas-api-tls") pod "maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt" (UID: "aaae968e-dd25-4c4c-8384-0b49e7137283") : secret "maas-api-e2e-ait-d4157b7d-serving-cert" not found
Jul 27 22:12:55.693162 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.693138    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae968e-dd25-4c4c-8384-0b49e7137283-openshift-service-ca\") pod \"maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt\" (UID: \"aaae968e-dd25-4c4c-8384-0b49e7137283\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:12:55.702397 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:55.702372    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dms6h\" (UniqueName: \"kubernetes.io/projected/aaae968e-dd25-4c4c-8384-0b49e7137283-kube-api-access-dms6h\") pod \"maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt\" (UID: \"aaae968e-dd25-4c4c-8384-0b49e7137283\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:12:56.196553 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.196518    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaae968e-dd25-4c4c-8384-0b49e7137283-maas-api-tls\") pod \"maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt\" (UID: \"aaae968e-dd25-4c4c-8384-0b49e7137283\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:12:56.198969 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.198951    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaae968e-dd25-4c4c-8384-0b49e7137283-maas-api-tls\") pod \"maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt\" (UID: \"aaae968e-dd25-4c4c-8384-0b49e7137283\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:12:56.392631 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.392600    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:12:56.457612 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.457530    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"]
Jul 27 22:12:56.462875 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.462853    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"
Jul 27 22:12:56.466684 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.466652    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d4157b7d\""
Jul 27 22:12:56.466871 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.466824    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d4157b7d-dockercfg-chrq5\""
Jul 27 22:12:56.472242 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.471954    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"]
Jul 27 22:12:56.520787 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.520750    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"]
Jul 27 22:12:56.525028 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.524996    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"
Jul 27 22:12:56.542715 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.542686    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"]
Jul 27 22:12:56.554666 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.554642    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"]
Jul 27 22:12:56.557530 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:12:56.557503    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaae968e_dd25_4c4c_8384_0b49e7137283.slice/crio-ffea1119a7292c8cd916c6e86d5796e4fae01f15f31b936c04d87c3752d9cb0a WatchSource:0}: Error finding container ffea1119a7292c8cd916c6e86d5796e4fae01f15f31b936c04d87c3752d9cb0a: Status 404 returned error can't find the container with id ffea1119a7292c8cd916c6e86d5796e4fae01f15f31b936c04d87c3752d9cb0a
Jul 27 22:12:56.599156 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.599117    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-672g8\" (UniqueName: \"kubernetes.io/projected/6df04522-23af-4d40-8359-05382e23fac0-kube-api-access-672g8\") pod \"payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m\" (UID: \"6df04522-23af-4d40-8359-05382e23fac0\") " pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"
Jul 27 22:12:56.599356 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.599163    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761063cb-4a94-456a-8473-21d5137c28ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s\" (UID: \"761063cb-4a94-456a-8473-21d5137c28ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"
Jul 27 22:12:56.599356 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.599192    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2z8f\" (UniqueName: \"kubernetes.io/projected/761063cb-4a94-456a-8473-21d5137c28ef-kube-api-access-p2z8f\") pod \"payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s\" (UID: \"761063cb-4a94-456a-8473-21d5137c28ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"
Jul 27 22:12:56.599356 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.599281    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df04522-23af-4d40-8359-05382e23fac0-plugins-config-volume\") pod \"payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m\" (UID: \"6df04522-23af-4d40-8359-05382e23fac0\") " pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"
Jul 27 22:12:56.699882 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.699851    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672g8\" (UniqueName: \"kubernetes.io/projected/6df04522-23af-4d40-8359-05382e23fac0-kube-api-access-672g8\") pod \"payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m\" (UID: \"6df04522-23af-4d40-8359-05382e23fac0\") " pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"
Jul 27 22:12:56.700057 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.699893    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761063cb-4a94-456a-8473-21d5137c28ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s\" (UID: \"761063cb-4a94-456a-8473-21d5137c28ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"
Jul 27 22:12:56.700057 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.699921    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2z8f\" (UniqueName: \"kubernetes.io/projected/761063cb-4a94-456a-8473-21d5137c28ef-kube-api-access-p2z8f\") pod \"payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s\" (UID: \"761063cb-4a94-456a-8473-21d5137c28ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"
Jul 27 22:12:56.700180 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.700068    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df04522-23af-4d40-8359-05382e23fac0-plugins-config-volume\") pod \"payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m\" (UID: \"6df04522-23af-4d40-8359-05382e23fac0\") " pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"
Jul 27 22:12:56.700668 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.700643    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761063cb-4a94-456a-8473-21d5137c28ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s\" (UID: \"761063cb-4a94-456a-8473-21d5137c28ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"
Jul 27 22:12:56.700768 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.700746    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df04522-23af-4d40-8359-05382e23fac0-plugins-config-volume\") pod \"payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m\" (UID: \"6df04522-23af-4d40-8359-05382e23fac0\") " pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"
Jul 27 22:12:56.708838 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.708781    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-672g8\" (UniqueName: \"kubernetes.io/projected/6df04522-23af-4d40-8359-05382e23fac0-kube-api-access-672g8\") pod \"payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m\" (UID: \"6df04522-23af-4d40-8359-05382e23fac0\") " pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"
Jul 27 22:12:56.708838 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.708783    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2z8f\" (UniqueName: \"kubernetes.io/projected/761063cb-4a94-456a-8473-21d5137c28ef-kube-api-access-p2z8f\") pod \"payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s\" (UID: \"761063cb-4a94-456a-8473-21d5137c28ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"
Jul 27 22:12:56.776386 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.776348    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"
Jul 27 22:12:56.792074 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.792036    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt" event={"ID":"aaae968e-dd25-4c4c-8384-0b49e7137283","Type":"ContainerStarted","Data":"ffea1119a7292c8cd916c6e86d5796e4fae01f15f31b936c04d87c3752d9cb0a"}
Jul 27 22:12:56.846639 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.846605    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"
Jul 27 22:12:56.936524 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:56.936499    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"]
Jul 27 22:12:56.938950 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:12:56.938917    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761063cb_4a94_456a_8473_21d5137c28ef.slice/crio-b3a99d9917832f69c7ce870f1b317d5b83a293de3c9c8d9f8a3ab9a30e6d3b18 WatchSource:0}: Error finding container b3a99d9917832f69c7ce870f1b317d5b83a293de3c9c8d9f8a3ab9a30e6d3b18: Status 404 returned error can't find the container with id b3a99d9917832f69c7ce870f1b317d5b83a293de3c9c8d9f8a3ab9a30e6d3b18
Jul 27 22:12:57.011183 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:57.011118    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"]
Jul 27 22:12:57.013393 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:12:57.013361    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df04522_23af_4d40_8359_05382e23fac0.slice/crio-38144293e1a51cce402b96b7b1b652eb10fe50db63a1ea6a0282b7604939e999 WatchSource:0}: Error finding container 38144293e1a51cce402b96b7b1b652eb10fe50db63a1ea6a0282b7604939e999: Status 404 returned error can't find the container with id 38144293e1a51cce402b96b7b1b652eb10fe50db63a1ea6a0282b7604939e999
Jul 27 22:12:57.799063 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:57.799009    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s" event={"ID":"761063cb-4a94-456a-8473-21d5137c28ef","Type":"ContainerStarted","Data":"4306b78f3c38493f04dab477e8c9211dbb2cc63f4aae24c6fd97f14b0956918f"}
Jul 27 22:12:57.799546 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:57.799087    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"
Jul 27 22:12:57.799546 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:57.799111    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s" event={"ID":"761063cb-4a94-456a-8473-21d5137c28ef","Type":"ContainerStarted","Data":"b3a99d9917832f69c7ce870f1b317d5b83a293de3c9c8d9f8a3ab9a30e6d3b18"}
Jul 27 22:12:57.800737 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:57.800708    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m" event={"ID":"6df04522-23af-4d40-8359-05382e23fac0","Type":"ContainerStarted","Data":"846d61bb252e0c3d7eea0413d77b6a9cd116dde064e224375379667b189e168c"}
Jul 27 22:12:57.800894 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:57.800763    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"
Jul 27 22:12:57.800894 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:57.800788    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m" event={"ID":"6df04522-23af-4d40-8359-05382e23fac0","Type":"ContainerStarted","Data":"38144293e1a51cce402b96b7b1b652eb10fe50db63a1ea6a0282b7604939e999"}
Jul 27 22:12:57.802413 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:57.802389    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt" event={"ID":"aaae968e-dd25-4c4c-8384-0b49e7137283","Type":"ContainerStarted","Data":"25f8cf2a6781dfcbf71d4084c95811e8c5f7b1295dca5949efb3e6f13d33a5a1"}
Jul 27 22:12:57.802569 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:57.802530    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:12:57.816934 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:57.816892    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s" podStartSLOduration=1.816879433 podStartE2EDuration="1.816879433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:12:57.816003936 +0000 UTC m=+1662.627351308" watchObservedRunningTime="2026-07-27 22:12:57.816879433 +0000 UTC m=+1662.628226805"
Jul 27 22:12:57.841941 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:57.841887    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m" podStartSLOduration=1.8418699219999999 podStartE2EDuration="1.841869922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:12:57.839930088 +0000 UTC m=+1662.651277460" watchObservedRunningTime="2026-07-27 22:12:57.841869922 +0000 UTC m=+1662.653217294"
Jul 27 22:12:57.858964 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:12:57.858912    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt" podStartSLOduration=2.525172369 podStartE2EDuration="2.858895608s" totalImagesPullingTime="333.723239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:12:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:12:57.857732189 +0000 UTC m=+1662.669079562" watchObservedRunningTime="2026-07-27 22:12:57.858895608 +0000 UTC m=+1662.670242981"
Jul 27 22:13:03.814938 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:03.814906    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:13:08.809411 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:08.809345    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"
Jul 27 22:13:08.809851 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:08.809424    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"
Jul 27 22:13:31.599549 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.599514    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"]
Jul 27 22:13:31.600194 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.600143    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt" podUID="aaae968e-dd25-4c4c-8384-0b49e7137283" containerName="maas-api" containerID="cri-o://25f8cf2a6781dfcbf71d4084c95811e8c5f7b1295dca5949efb3e6f13d33a5a1" gracePeriod=30
Jul 27 22:13:31.641121 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.641082    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"]
Jul 27 22:13:31.641465 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.641415    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m" podUID="6df04522-23af-4d40-8359-05382e23fac0" containerName="payload-processing" containerID="cri-o://846d61bb252e0c3d7eea0413d77b6a9cd116dde064e224375379667b189e168c" gracePeriod=30
Jul 27 22:13:31.651632 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.651517    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"]
Jul 27 22:13:31.651966 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.651860    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s" podUID="761063cb-4a94-456a-8473-21d5137c28ef" containerName="payload-pre-processing" containerID="cri-o://4306b78f3c38493f04dab477e8c9211dbb2cc63f4aae24c6fd97f14b0956918f" gracePeriod=30
Jul 27 22:13:31.918866 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.918839    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"
Jul 27 22:13:31.956240 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.956203    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:13:31.959421 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.959396    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"
Jul 27 22:13:31.976435 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.976405    2549 generic.go:356] "Generic (PLEG): container finished" podID="6df04522-23af-4d40-8359-05382e23fac0" containerID="846d61bb252e0c3d7eea0413d77b6a9cd116dde064e224375379667b189e168c" exitCode=0
Jul 27 22:13:31.976580 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.976449    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"
Jul 27 22:13:31.976580 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.976483    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m" event={"ID":"6df04522-23af-4d40-8359-05382e23fac0","Type":"ContainerDied","Data":"846d61bb252e0c3d7eea0413d77b6a9cd116dde064e224375379667b189e168c"}
Jul 27 22:13:31.976580 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.976539    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m" event={"ID":"6df04522-23af-4d40-8359-05382e23fac0","Type":"ContainerDied","Data":"38144293e1a51cce402b96b7b1b652eb10fe50db63a1ea6a0282b7604939e999"}
Jul 27 22:13:31.976580 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.976568    2549 scope.go:117] "RemoveContainer" containerID="846d61bb252e0c3d7eea0413d77b6a9cd116dde064e224375379667b189e168c"
Jul 27 22:13:31.978091 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.978059    2549 generic.go:356] "Generic (PLEG): container finished" podID="aaae968e-dd25-4c4c-8384-0b49e7137283" containerID="25f8cf2a6781dfcbf71d4084c95811e8c5f7b1295dca5949efb3e6f13d33a5a1" exitCode=0
Jul 27 22:13:31.978265 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.978213    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt" event={"ID":"aaae968e-dd25-4c4c-8384-0b49e7137283","Type":"ContainerDied","Data":"25f8cf2a6781dfcbf71d4084c95811e8c5f7b1295dca5949efb3e6f13d33a5a1"}
Jul 27 22:13:31.978354 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.978338    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"
Jul 27 22:13:31.978594 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.978572    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt" event={"ID":"aaae968e-dd25-4c4c-8384-0b49e7137283","Type":"ContainerDied","Data":"ffea1119a7292c8cd916c6e86d5796e4fae01f15f31b936c04d87c3752d9cb0a"}
Jul 27 22:13:31.980213 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.979924    2549 generic.go:356] "Generic (PLEG): container finished" podID="761063cb-4a94-456a-8473-21d5137c28ef" containerID="4306b78f3c38493f04dab477e8c9211dbb2cc63f4aae24c6fd97f14b0956918f" exitCode=0
Jul 27 22:13:31.980213 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.980027    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s" event={"ID":"761063cb-4a94-456a-8473-21d5137c28ef","Type":"ContainerDied","Data":"4306b78f3c38493f04dab477e8c9211dbb2cc63f4aae24c6fd97f14b0956918f"}
Jul 27 22:13:31.980213 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.980116    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s" event={"ID":"761063cb-4a94-456a-8473-21d5137c28ef","Type":"ContainerDied","Data":"b3a99d9917832f69c7ce870f1b317d5b83a293de3c9c8d9f8a3ab9a30e6d3b18"}
Jul 27 22:13:31.980213 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.980192    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"
Jul 27 22:13:31.988805 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.988779    2549 scope.go:117] "RemoveContainer" containerID="846d61bb252e0c3d7eea0413d77b6a9cd116dde064e224375379667b189e168c"
Jul 27 22:13:31.989178 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:13:31.989153    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846d61bb252e0c3d7eea0413d77b6a9cd116dde064e224375379667b189e168c\": container with ID starting with 846d61bb252e0c3d7eea0413d77b6a9cd116dde064e224375379667b189e168c not found: ID does not exist" containerID="846d61bb252e0c3d7eea0413d77b6a9cd116dde064e224375379667b189e168c"
Jul 27 22:13:31.989290 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.989189    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846d61bb252e0c3d7eea0413d77b6a9cd116dde064e224375379667b189e168c"} err="failed to get container status \"846d61bb252e0c3d7eea0413d77b6a9cd116dde064e224375379667b189e168c\": rpc error: code = NotFound desc = could not find container \"846d61bb252e0c3d7eea0413d77b6a9cd116dde064e224375379667b189e168c\": container with ID starting with 846d61bb252e0c3d7eea0413d77b6a9cd116dde064e224375379667b189e168c not found: ID does not exist"
Jul 27 22:13:31.989290 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.989214    2549 scope.go:117] "RemoveContainer" containerID="25f8cf2a6781dfcbf71d4084c95811e8c5f7b1295dca5949efb3e6f13d33a5a1"
Jul 27 22:13:31.999381 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.999286    2549 scope.go:117] "RemoveContainer" containerID="25f8cf2a6781dfcbf71d4084c95811e8c5f7b1295dca5949efb3e6f13d33a5a1"
Jul 27 22:13:31.999710 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:13:31.999669    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f8cf2a6781dfcbf71d4084c95811e8c5f7b1295dca5949efb3e6f13d33a5a1\": container with ID starting with 25f8cf2a6781dfcbf71d4084c95811e8c5f7b1295dca5949efb3e6f13d33a5a1 not found: ID does not exist" containerID="25f8cf2a6781dfcbf71d4084c95811e8c5f7b1295dca5949efb3e6f13d33a5a1"
Jul 27 22:13:31.999803 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.999710    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f8cf2a6781dfcbf71d4084c95811e8c5f7b1295dca5949efb3e6f13d33a5a1"} err="failed to get container status \"25f8cf2a6781dfcbf71d4084c95811e8c5f7b1295dca5949efb3e6f13d33a5a1\": rpc error: code = NotFound desc = could not find container \"25f8cf2a6781dfcbf71d4084c95811e8c5f7b1295dca5949efb3e6f13d33a5a1\": container with ID starting with 25f8cf2a6781dfcbf71d4084c95811e8c5f7b1295dca5949efb3e6f13d33a5a1 not found: ID does not exist"
Jul 27 22:13:31.999803 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:31.999738    2549 scope.go:117] "RemoveContainer" containerID="4306b78f3c38493f04dab477e8c9211dbb2cc63f4aae24c6fd97f14b0956918f"
Jul 27 22:13:32.008493 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.008472    2549 scope.go:117] "RemoveContainer" containerID="4306b78f3c38493f04dab477e8c9211dbb2cc63f4aae24c6fd97f14b0956918f"
Jul 27 22:13:32.008790 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:13:32.008771    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4306b78f3c38493f04dab477e8c9211dbb2cc63f4aae24c6fd97f14b0956918f\": container with ID starting with 4306b78f3c38493f04dab477e8c9211dbb2cc63f4aae24c6fd97f14b0956918f not found: ID does not exist" containerID="4306b78f3c38493f04dab477e8c9211dbb2cc63f4aae24c6fd97f14b0956918f"
Jul 27 22:13:32.008850 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.008799    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4306b78f3c38493f04dab477e8c9211dbb2cc63f4aae24c6fd97f14b0956918f"} err="failed to get container status \"4306b78f3c38493f04dab477e8c9211dbb2cc63f4aae24c6fd97f14b0956918f\": rpc error: code = NotFound desc = could not find container \"4306b78f3c38493f04dab477e8c9211dbb2cc63f4aae24c6fd97f14b0956918f\": container with ID starting with 4306b78f3c38493f04dab477e8c9211dbb2cc63f4aae24c6fd97f14b0956918f not found: ID does not exist"
Jul 27 22:13:32.035586 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.035555    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dms6h\" (UniqueName: \"kubernetes.io/projected/aaae968e-dd25-4c4c-8384-0b49e7137283-kube-api-access-dms6h\") pod \"aaae968e-dd25-4c4c-8384-0b49e7137283\" (UID: \"aaae968e-dd25-4c4c-8384-0b49e7137283\") "
Jul 27 22:13:32.035586 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.035589    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2z8f\" (UniqueName: \"kubernetes.io/projected/761063cb-4a94-456a-8473-21d5137c28ef-kube-api-access-p2z8f\") pod \"761063cb-4a94-456a-8473-21d5137c28ef\" (UID: \"761063cb-4a94-456a-8473-21d5137c28ef\") "
Jul 27 22:13:32.035820 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.035634    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761063cb-4a94-456a-8473-21d5137c28ef-plugins-config-volume\") pod \"761063cb-4a94-456a-8473-21d5137c28ef\" (UID: \"761063cb-4a94-456a-8473-21d5137c28ef\") "
Jul 27 22:13:32.035820 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.035659    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-672g8\" (UniqueName: \"kubernetes.io/projected/6df04522-23af-4d40-8359-05382e23fac0-kube-api-access-672g8\") pod \"6df04522-23af-4d40-8359-05382e23fac0\" (UID: \"6df04522-23af-4d40-8359-05382e23fac0\") "
Jul 27 22:13:32.035820 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.035688    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae968e-dd25-4c4c-8384-0b49e7137283-openshift-service-ca\") pod \"aaae968e-dd25-4c4c-8384-0b49e7137283\" (UID: \"aaae968e-dd25-4c4c-8384-0b49e7137283\") "
Jul 27 22:13:32.035820 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.035704    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaae968e-dd25-4c4c-8384-0b49e7137283-maas-api-tls\") pod \"aaae968e-dd25-4c4c-8384-0b49e7137283\" (UID: \"aaae968e-dd25-4c4c-8384-0b49e7137283\") "
Jul 27 22:13:32.035820 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.035740    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df04522-23af-4d40-8359-05382e23fac0-plugins-config-volume\") pod \"6df04522-23af-4d40-8359-05382e23fac0\" (UID: \"6df04522-23af-4d40-8359-05382e23fac0\") "
Jul 27 22:13:32.036135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.036085    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761063cb-4a94-456a-8473-21d5137c28ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "761063cb-4a94-456a-8473-21d5137c28ef" (UID: "761063cb-4a94-456a-8473-21d5137c28ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:13:32.036135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.036096    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaae968e-dd25-4c4c-8384-0b49e7137283-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aaae968e-dd25-4c4c-8384-0b49e7137283" (UID: "aaae968e-dd25-4c4c-8384-0b49e7137283"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:13:32.036265 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.036177    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df04522-23af-4d40-8359-05382e23fac0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6df04522-23af-4d40-8359-05382e23fac0" (UID: "6df04522-23af-4d40-8359-05382e23fac0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:13:32.036653 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.036585    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae968e-dd25-4c4c-8384-0b49e7137283-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:13:32.036748 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.036658    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df04522-23af-4d40-8359-05382e23fac0-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:13:32.036748 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.036672    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761063cb-4a94-456a-8473-21d5137c28ef-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:13:32.037904 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.037864    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaae968e-dd25-4c4c-8384-0b49e7137283-kube-api-access-dms6h" (OuterVolumeSpecName: "kube-api-access-dms6h") pod "aaae968e-dd25-4c4c-8384-0b49e7137283" (UID: "aaae968e-dd25-4c4c-8384-0b49e7137283"). InnerVolumeSpecName "kube-api-access-dms6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:13:32.037995 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.037943    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaae968e-dd25-4c4c-8384-0b49e7137283-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aaae968e-dd25-4c4c-8384-0b49e7137283" (UID: "aaae968e-dd25-4c4c-8384-0b49e7137283"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:13:32.037995 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.037959    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761063cb-4a94-456a-8473-21d5137c28ef-kube-api-access-p2z8f" (OuterVolumeSpecName: "kube-api-access-p2z8f") pod "761063cb-4a94-456a-8473-21d5137c28ef" (UID: "761063cb-4a94-456a-8473-21d5137c28ef"). InnerVolumeSpecName "kube-api-access-p2z8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:13:32.038080 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.037992    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df04522-23af-4d40-8359-05382e23fac0-kube-api-access-672g8" (OuterVolumeSpecName: "kube-api-access-672g8") pod "6df04522-23af-4d40-8359-05382e23fac0" (UID: "6df04522-23af-4d40-8359-05382e23fac0"). InnerVolumeSpecName "kube-api-access-672g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:13:32.137770 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.137681    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dms6h\" (UniqueName: \"kubernetes.io/projected/aaae968e-dd25-4c4c-8384-0b49e7137283-kube-api-access-dms6h\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:13:32.137770 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.137711    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2z8f\" (UniqueName: \"kubernetes.io/projected/761063cb-4a94-456a-8473-21d5137c28ef-kube-api-access-p2z8f\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:13:32.137770 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.137720    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-672g8\" (UniqueName: \"kubernetes.io/projected/6df04522-23af-4d40-8359-05382e23fac0-kube-api-access-672g8\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:13:32.137770 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.137729    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaae968e-dd25-4c4c-8384-0b49e7137283-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:13:32.303194 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.303163    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"]
Jul 27 22:13:32.307289 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.307258    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d4157b7d-6bdc764f6-6k77m"]
Jul 27 22:13:32.318776 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.318744    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"]
Jul 27 22:13:32.323612 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.323589    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d4157b7d-6b4fd7d8bd-hpr9s"]
Jul 27 22:13:32.334757 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.334733    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"]
Jul 27 22:13:32.340367 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:32.340340    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d4157b7d-66bdf546bc-7p6nt"]
Jul 27 22:13:33.700779 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:33.700744    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df04522-23af-4d40-8359-05382e23fac0" path="/var/lib/kubelet/pods/6df04522-23af-4d40-8359-05382e23fac0/volumes"
Jul 27 22:13:33.701146 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:33.701110    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="761063cb-4a94-456a-8473-21d5137c28ef" path="/var/lib/kubelet/pods/761063cb-4a94-456a-8473-21d5137c28ef/volumes"
Jul 27 22:13:33.701507 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:33.701491    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaae968e-dd25-4c4c-8384-0b49e7137283" path="/var/lib/kubelet/pods/aaae968e-dd25-4c4c-8384-0b49e7137283/volumes"
Jul 27 22:13:40.674078 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.673999    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"]
Jul 27 22:13:40.674473 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.674400    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761063cb-4a94-456a-8473-21d5137c28ef" containerName="payload-pre-processing"
Jul 27 22:13:40.674473 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.674412    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="761063cb-4a94-456a-8473-21d5137c28ef" containerName="payload-pre-processing"
Jul 27 22:13:40.674473 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.674421    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df04522-23af-4d40-8359-05382e23fac0" containerName="payload-processing"
Jul 27 22:13:40.674473 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.674427    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df04522-23af-4d40-8359-05382e23fac0" containerName="payload-processing"
Jul 27 22:13:40.674473 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.674435    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaae968e-dd25-4c4c-8384-0b49e7137283" containerName="maas-api"
Jul 27 22:13:40.674473 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.674441    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaae968e-dd25-4c4c-8384-0b49e7137283" containerName="maas-api"
Jul 27 22:13:40.674658 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.674502    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df04522-23af-4d40-8359-05382e23fac0" containerName="payload-processing"
Jul 27 22:13:40.674658 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.674509    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="761063cb-4a94-456a-8473-21d5137c28ef" containerName="payload-pre-processing"
Jul 27 22:13:40.674658 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.674516    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaae968e-dd25-4c4c-8384-0b49e7137283" containerName="maas-api"
Jul 27 22:13:40.679349 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.679328    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:13:40.683047 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.683031    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-78fe627e-serving-cert\""
Jul 27 22:13:40.700669 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.700641    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"]
Jul 27 22:13:40.818686 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.818649    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf13facf-f5b9-45d1-8cdd-6efe589ef917-openshift-service-ca\") pod \"maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn\" (UID: \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:13:40.818863 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.818696    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd7rz\" (UniqueName: \"kubernetes.io/projected/bf13facf-f5b9-45d1-8cdd-6efe589ef917-kube-api-access-cd7rz\") pod \"maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn\" (UID: \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:13:40.818863 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.818724    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf13facf-f5b9-45d1-8cdd-6efe589ef917-maas-api-tls\") pod \"maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn\" (UID: \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:13:40.919530 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.919492    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf13facf-f5b9-45d1-8cdd-6efe589ef917-openshift-service-ca\") pod \"maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn\" (UID: \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:13:40.919678 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.919539    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd7rz\" (UniqueName: \"kubernetes.io/projected/bf13facf-f5b9-45d1-8cdd-6efe589ef917-kube-api-access-cd7rz\") pod \"maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn\" (UID: \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:13:40.919678 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.919565    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf13facf-f5b9-45d1-8cdd-6efe589ef917-maas-api-tls\") pod \"maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn\" (UID: \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:13:40.919678 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:13:40.919663    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-serving-cert: secret "maas-api-e2e-derive-78fe627e-serving-cert" not found
Jul 27 22:13:40.919815 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:13:40.919718    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf13facf-f5b9-45d1-8cdd-6efe589ef917-maas-api-tls podName:bf13facf-f5b9-45d1-8cdd-6efe589ef917 nodeName:}" failed. No retries permitted until 2026-07-27 22:13:41.419701325 +0000 UTC m=+1706.231048675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf13facf-f5b9-45d1-8cdd-6efe589ef917-maas-api-tls") pod "maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn" (UID: "bf13facf-f5b9-45d1-8cdd-6efe589ef917") : secret "maas-api-e2e-derive-78fe627e-serving-cert" not found
Jul 27 22:13:40.920175 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.920155    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf13facf-f5b9-45d1-8cdd-6efe589ef917-openshift-service-ca\") pod \"maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn\" (UID: \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:13:40.938162 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:40.938099    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd7rz\" (UniqueName: \"kubernetes.io/projected/bf13facf-f5b9-45d1-8cdd-6efe589ef917-kube-api-access-cd7rz\") pod \"maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn\" (UID: \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:13:41.425032 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.424982    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf13facf-f5b9-45d1-8cdd-6efe589ef917-maas-api-tls\") pod \"maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn\" (UID: \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:13:41.427835 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.427805    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf13facf-f5b9-45d1-8cdd-6efe589ef917-maas-api-tls\") pod \"maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn\" (UID: \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:13:41.497818 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.497777    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"]
Jul 27 22:13:41.501480 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.501460    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"
Jul 27 22:13:41.504558 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.504534    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-78fe627e-dockercfg-zplkr\""
Jul 27 22:13:41.504716 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.504654    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-78fe627e\""
Jul 27 22:13:41.513539 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.513515    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"]
Jul 27 22:13:41.525567 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.525532    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"]
Jul 27 22:13:41.528949 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.528930    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"
Jul 27 22:13:41.539358 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.539335    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"]
Jul 27 22:13:41.589499 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.589463    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:13:41.626896 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.626854    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvc29\" (UniqueName: \"kubernetes.io/projected/fb14c347-2128-45a8-92f7-b9a17fb5989e-kube-api-access-hvc29\") pod \"payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z\" (UID: \"fb14c347-2128-45a8-92f7-b9a17fb5989e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"
Jul 27 22:13:41.627096 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.626910    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rjj6\" (UniqueName: \"kubernetes.io/projected/3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d-kube-api-access-6rjj6\") pod \"payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25\" (UID: \"3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d\") " pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"
Jul 27 22:13:41.627096 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.627022    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d-plugins-config-volume\") pod \"payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25\" (UID: \"3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d\") " pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"
Jul 27 22:13:41.627256 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.627118    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb14c347-2128-45a8-92f7-b9a17fb5989e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z\" (UID: \"fb14c347-2128-45a8-92f7-b9a17fb5989e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"
Jul 27 22:13:41.728467 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.728434    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d-plugins-config-volume\") pod \"payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25\" (UID: \"3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d\") " pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"
Jul 27 22:13:41.728984 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.728497    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb14c347-2128-45a8-92f7-b9a17fb5989e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z\" (UID: \"fb14c347-2128-45a8-92f7-b9a17fb5989e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"
Jul 27 22:13:41.728984 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.728532    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvc29\" (UniqueName: \"kubernetes.io/projected/fb14c347-2128-45a8-92f7-b9a17fb5989e-kube-api-access-hvc29\") pod \"payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z\" (UID: \"fb14c347-2128-45a8-92f7-b9a17fb5989e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"
Jul 27 22:13:41.728984 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.728560    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rjj6\" (UniqueName: \"kubernetes.io/projected/3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d-kube-api-access-6rjj6\") pod \"payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25\" (UID: \"3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d\") " pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"
Jul 27 22:13:41.729191 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.729176    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d-plugins-config-volume\") pod \"payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25\" (UID: \"3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d\") " pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"
Jul 27 22:13:41.729390 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.729368    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb14c347-2128-45a8-92f7-b9a17fb5989e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z\" (UID: \"fb14c347-2128-45a8-92f7-b9a17fb5989e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"
Jul 27 22:13:41.732972 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.732949    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"]
Jul 27 22:13:41.737197 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:13:41.737165    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf13facf_f5b9_45d1_8cdd_6efe589ef917.slice/crio-3a4e326fa458ed0e5f627bd0ca89c22385f04142568f4cbf9a81f31d18cf9c60 WatchSource:0}: Error finding container 3a4e326fa458ed0e5f627bd0ca89c22385f04142568f4cbf9a81f31d18cf9c60: Status 404 returned error can't find the container with id 3a4e326fa458ed0e5f627bd0ca89c22385f04142568f4cbf9a81f31d18cf9c60
Jul 27 22:13:41.738483 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.738452    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvc29\" (UniqueName: \"kubernetes.io/projected/fb14c347-2128-45a8-92f7-b9a17fb5989e-kube-api-access-hvc29\") pod \"payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z\" (UID: \"fb14c347-2128-45a8-92f7-b9a17fb5989e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"
Jul 27 22:13:41.738593 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.738564    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rjj6\" (UniqueName: \"kubernetes.io/projected/3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d-kube-api-access-6rjj6\") pod \"payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25\" (UID: \"3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d\") " pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"
Jul 27 22:13:41.813244 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.813195    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"
Jul 27 22:13:41.840823 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.840795    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"
Jul 27 22:13:41.976136 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:41.976110    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"]
Jul 27 22:13:41.977677 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:13:41.977633    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef8d574_b1e0_4f4e_b0d5_e191b7d4a52d.slice/crio-ea466c540f5d3fda1b28a41edba5ebab074dcbb5ae3e87dc94fd85598d413a11 WatchSource:0}: Error finding container ea466c540f5d3fda1b28a41edba5ebab074dcbb5ae3e87dc94fd85598d413a11: Status 404 returned error can't find the container with id ea466c540f5d3fda1b28a41edba5ebab074dcbb5ae3e87dc94fd85598d413a11
Jul 27 22:13:42.004020 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:42.003880    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"]
Jul 27 22:13:42.006395 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:13:42.006369    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb14c347_2128_45a8_92f7_b9a17fb5989e.slice/crio-dcc270b9ccb1fa1383a90b9076dbdbf8bbf0dba557390b8507a4b360dea80721 WatchSource:0}: Error finding container dcc270b9ccb1fa1383a90b9076dbdbf8bbf0dba557390b8507a4b360dea80721: Status 404 returned error can't find the container with id dcc270b9ccb1fa1383a90b9076dbdbf8bbf0dba557390b8507a4b360dea80721
Jul 27 22:13:42.035799 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:42.035765    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z" event={"ID":"fb14c347-2128-45a8-92f7-b9a17fb5989e","Type":"ContainerStarted","Data":"dcc270b9ccb1fa1383a90b9076dbdbf8bbf0dba557390b8507a4b360dea80721"}
Jul 27 22:13:42.037119 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:42.037093    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25" event={"ID":"3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d","Type":"ContainerStarted","Data":"ea466c540f5d3fda1b28a41edba5ebab074dcbb5ae3e87dc94fd85598d413a11"}
Jul 27 22:13:42.038343 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:42.038309    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn" event={"ID":"bf13facf-f5b9-45d1-8cdd-6efe589ef917","Type":"ContainerStarted","Data":"3a4e326fa458ed0e5f627bd0ca89c22385f04142568f4cbf9a81f31d18cf9c60"}
Jul 27 22:13:43.045150 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:43.045106    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25" event={"ID":"3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d","Type":"ContainerStarted","Data":"411729dcc8592d4b4cf8869ad5f2eb7c01d947d584ec3b488e09c6fb90c2f53c"}
Jul 27 22:13:43.045633 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:43.045172    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"
Jul 27 22:13:43.046810 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:43.046781    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn" event={"ID":"bf13facf-f5b9-45d1-8cdd-6efe589ef917","Type":"ContainerStarted","Data":"0b1c39ca3473b87464d92a51c2cfb8b2d8aa6eaa578d57bd08481b091988443f"}
Jul 27 22:13:43.046922 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:43.046839    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:13:43.048216 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:43.048191    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z" event={"ID":"fb14c347-2128-45a8-92f7-b9a17fb5989e","Type":"ContainerStarted","Data":"868a30f01c3462ef818f32eb4d61febd74f83f2038415b9cb4642987201b94a7"}
Jul 27 22:13:43.048330 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:43.048280    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"
Jul 27 22:13:43.066356 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:43.066278    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25" podStartSLOduration=2.066261234 podStartE2EDuration="2.066261234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:13:43.065569712 +0000 UTC m=+1707.876917085" watchObservedRunningTime="2026-07-27 22:13:43.066261234 +0000 UTC m=+1707.877608605"
Jul 27 22:13:43.086947 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:43.086896    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn" podStartSLOduration=2.674806884 podStartE2EDuration="3.086880704s" totalImagesPullingTime="412.07382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:13:43.085681446 +0000 UTC m=+1707.897028822" watchObservedRunningTime="2026-07-27 22:13:43.086880704 +0000 UTC m=+1707.898228115"
Jul 27 22:13:43.107975 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:43.107927    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z" podStartSLOduration=2.107911242 podStartE2EDuration="2.107911242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:13:43.106504578 +0000 UTC m=+1707.917851951" watchObservedRunningTime="2026-07-27 22:13:43.107911242 +0000 UTC m=+1707.919258648"
Jul 27 22:13:49.059755 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:49.059725    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:13:54.055405 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:54.055360    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"
Jul 27 22:13:54.055819 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:13:54.055612    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"
Jul 27 22:14:15.984280 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:15.984239    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"]
Jul 27 22:14:15.984829 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:15.984509    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn" podUID="bf13facf-f5b9-45d1-8cdd-6efe589ef917" containerName="maas-api" containerID="cri-o://0b1c39ca3473b87464d92a51c2cfb8b2d8aa6eaa578d57bd08481b091988443f" gracePeriod=30
Jul 27 22:14:16.043177 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.042983    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"]
Jul 27 22:14:16.043390 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.043361    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25" podUID="3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d" containerName="payload-processing" containerID="cri-o://411729dcc8592d4b4cf8869ad5f2eb7c01d947d584ec3b488e09c6fb90c2f53c" gracePeriod=30
Jul 27 22:14:16.051097 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.051063    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"]
Jul 27 22:14:16.051641 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.051382    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z" podUID="fb14c347-2128-45a8-92f7-b9a17fb5989e" containerName="payload-pre-processing" containerID="cri-o://868a30f01c3462ef818f32eb4d61febd74f83f2038415b9cb4642987201b94a7" gracePeriod=30
Jul 27 22:14:16.224870 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.224820    2549 generic.go:356] "Generic (PLEG): container finished" podID="3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d" containerID="411729dcc8592d4b4cf8869ad5f2eb7c01d947d584ec3b488e09c6fb90c2f53c" exitCode=0
Jul 27 22:14:16.225034 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.224925    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25" event={"ID":"3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d","Type":"ContainerDied","Data":"411729dcc8592d4b4cf8869ad5f2eb7c01d947d584ec3b488e09c6fb90c2f53c"}
Jul 27 22:14:16.227089 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.226867    2549 generic.go:356] "Generic (PLEG): container finished" podID="bf13facf-f5b9-45d1-8cdd-6efe589ef917" containerID="0b1c39ca3473b87464d92a51c2cfb8b2d8aa6eaa578d57bd08481b091988443f" exitCode=0
Jul 27 22:14:16.227089 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.227032    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn" event={"ID":"bf13facf-f5b9-45d1-8cdd-6efe589ef917","Type":"ContainerDied","Data":"0b1c39ca3473b87464d92a51c2cfb8b2d8aa6eaa578d57bd08481b091988443f"}
Jul 27 22:14:16.229055 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.229030    2549 generic.go:356] "Generic (PLEG): container finished" podID="fb14c347-2128-45a8-92f7-b9a17fb5989e" containerID="868a30f01c3462ef818f32eb4d61febd74f83f2038415b9cb4642987201b94a7" exitCode=0
Jul 27 22:14:16.229168 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.229094    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z" event={"ID":"fb14c347-2128-45a8-92f7-b9a17fb5989e","Type":"ContainerDied","Data":"868a30f01c3462ef818f32eb4d61febd74f83f2038415b9cb4642987201b94a7"}
Jul 27 22:14:16.337659 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.337635    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"
Jul 27 22:14:16.340619 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.340592    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:14:16.353151 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.353130    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"
Jul 27 22:14:16.434275 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.434210    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvc29\" (UniqueName: \"kubernetes.io/projected/fb14c347-2128-45a8-92f7-b9a17fb5989e-kube-api-access-hvc29\") pod \"fb14c347-2128-45a8-92f7-b9a17fb5989e\" (UID: \"fb14c347-2128-45a8-92f7-b9a17fb5989e\") "
Jul 27 22:14:16.434474 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.434307    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd7rz\" (UniqueName: \"kubernetes.io/projected/bf13facf-f5b9-45d1-8cdd-6efe589ef917-kube-api-access-cd7rz\") pod \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\" (UID: \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\") "
Jul 27 22:14:16.434474 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.434336    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rjj6\" (UniqueName: \"kubernetes.io/projected/3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d-kube-api-access-6rjj6\") pod \"3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d\" (UID: \"3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d\") "
Jul 27 22:14:16.434474 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.434374    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb14c347-2128-45a8-92f7-b9a17fb5989e-plugins-config-volume\") pod \"fb14c347-2128-45a8-92f7-b9a17fb5989e\" (UID: \"fb14c347-2128-45a8-92f7-b9a17fb5989e\") "
Jul 27 22:14:16.434474 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.434402    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf13facf-f5b9-45d1-8cdd-6efe589ef917-maas-api-tls\") pod \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\" (UID: \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\") "
Jul 27 22:14:16.434474 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.434438    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d-plugins-config-volume\") pod \"3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d\" (UID: \"3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d\") "
Jul 27 22:14:16.434757 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.434481    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf13facf-f5b9-45d1-8cdd-6efe589ef917-openshift-service-ca\") pod \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\" (UID: \"bf13facf-f5b9-45d1-8cdd-6efe589ef917\") "
Jul 27 22:14:16.434842 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.434801    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb14c347-2128-45a8-92f7-b9a17fb5989e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb14c347-2128-45a8-92f7-b9a17fb5989e" (UID: "fb14c347-2128-45a8-92f7-b9a17fb5989e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:14:16.434922 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.434882    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d" (UID: "3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:14:16.434922 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.434904    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf13facf-f5b9-45d1-8cdd-6efe589ef917-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf13facf-f5b9-45d1-8cdd-6efe589ef917" (UID: "bf13facf-f5b9-45d1-8cdd-6efe589ef917"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:14:16.436793 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.436768    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf13facf-f5b9-45d1-8cdd-6efe589ef917-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf13facf-f5b9-45d1-8cdd-6efe589ef917" (UID: "bf13facf-f5b9-45d1-8cdd-6efe589ef917"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:14:16.436793 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.436781    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf13facf-f5b9-45d1-8cdd-6efe589ef917-kube-api-access-cd7rz" (OuterVolumeSpecName: "kube-api-access-cd7rz") pod "bf13facf-f5b9-45d1-8cdd-6efe589ef917" (UID: "bf13facf-f5b9-45d1-8cdd-6efe589ef917"). InnerVolumeSpecName "kube-api-access-cd7rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:14:16.437010 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.436993    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb14c347-2128-45a8-92f7-b9a17fb5989e-kube-api-access-hvc29" (OuterVolumeSpecName: "kube-api-access-hvc29") pod "fb14c347-2128-45a8-92f7-b9a17fb5989e" (UID: "fb14c347-2128-45a8-92f7-b9a17fb5989e"). InnerVolumeSpecName "kube-api-access-hvc29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:14:16.437051 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.437018    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d-kube-api-access-6rjj6" (OuterVolumeSpecName: "kube-api-access-6rjj6") pod "3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d" (UID: "3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d"). InnerVolumeSpecName "kube-api-access-6rjj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:14:16.535381 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.535339    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rjj6\" (UniqueName: \"kubernetes.io/projected/3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d-kube-api-access-6rjj6\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:14:16.535381 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.535371    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb14c347-2128-45a8-92f7-b9a17fb5989e-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:14:16.535381 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.535381    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf13facf-f5b9-45d1-8cdd-6efe589ef917-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:14:16.535381 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.535389    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:14:16.535660 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.535399    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf13facf-f5b9-45d1-8cdd-6efe589ef917-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:14:16.535660 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.535408    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvc29\" (UniqueName: \"kubernetes.io/projected/fb14c347-2128-45a8-92f7-b9a17fb5989e-kube-api-access-hvc29\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:14:16.535660 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:16.535416    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd7rz\" (UniqueName: \"kubernetes.io/projected/bf13facf-f5b9-45d1-8cdd-6efe589ef917-kube-api-access-cd7rz\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:14:17.235644 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.235615    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"
Jul 27 22:14:17.236056 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.235662    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z" event={"ID":"fb14c347-2128-45a8-92f7-b9a17fb5989e","Type":"ContainerDied","Data":"dcc270b9ccb1fa1383a90b9076dbdbf8bbf0dba557390b8507a4b360dea80721"}
Jul 27 22:14:17.236056 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.235730    2549 scope.go:117] "RemoveContainer" containerID="868a30f01c3462ef818f32eb4d61febd74f83f2038415b9cb4642987201b94a7"
Jul 27 22:14:17.236997 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.236946    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"
Jul 27 22:14:17.236997 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.236959    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25" event={"ID":"3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d","Type":"ContainerDied","Data":"ea466c540f5d3fda1b28a41edba5ebab074dcbb5ae3e87dc94fd85598d413a11"}
Jul 27 22:14:17.238342 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.238326    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"
Jul 27 22:14:17.238453 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.238324    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn" event={"ID":"bf13facf-f5b9-45d1-8cdd-6efe589ef917","Type":"ContainerDied","Data":"3a4e326fa458ed0e5f627bd0ca89c22385f04142568f4cbf9a81f31d18cf9c60"}
Jul 27 22:14:17.245453 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.245435    2549 scope.go:117] "RemoveContainer" containerID="411729dcc8592d4b4cf8869ad5f2eb7c01d947d584ec3b488e09c6fb90c2f53c"
Jul 27 22:14:17.254123 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.254107    2549 scope.go:117] "RemoveContainer" containerID="0b1c39ca3473b87464d92a51c2cfb8b2d8aa6eaa578d57bd08481b091988443f"
Jul 27 22:14:17.265180 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.265157    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"]
Jul 27 22:14:17.271973 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.271947    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-78fe627e-595d75c7f4-7nbxn"]
Jul 27 22:14:17.284372 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.284345    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"]
Jul 27 22:14:17.288764 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.288743    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-78fe627e-5fddc8b4bd-n2c7z"]
Jul 27 22:14:17.301027 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.301003    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"]
Jul 27 22:14:17.305411 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.305390    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-78fe627e-869ddd848b-dqx25"]
Jul 27 22:14:17.700880 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.700836    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d" path="/var/lib/kubelet/pods/3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d/volumes"
Jul 27 22:14:17.701325 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.701306    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf13facf-f5b9-45d1-8cdd-6efe589ef917" path="/var/lib/kubelet/pods/bf13facf-f5b9-45d1-8cdd-6efe589ef917/volumes"
Jul 27 22:14:17.701679 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:14:17.701662    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb14c347-2128-45a8-92f7-b9a17fb5989e" path="/var/lib/kubelet/pods/fb14c347-2128-45a8-92f7-b9a17fb5989e/volumes"
Jul 27 22:15:15.716791 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:15:15.716763    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:15:15.719472 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:15:15.719450    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:16:03.102551 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.099551    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"]
Jul 27 22:16:03.102551 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.100469    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb14c347-2128-45a8-92f7-b9a17fb5989e" containerName="payload-pre-processing"
Jul 27 22:16:03.102551 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.100491    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb14c347-2128-45a8-92f7-b9a17fb5989e" containerName="payload-pre-processing"
Jul 27 22:16:03.102551 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.100519    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d" containerName="payload-processing"
Jul 27 22:16:03.102551 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.100528    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d" containerName="payload-processing"
Jul 27 22:16:03.102551 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.100558    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf13facf-f5b9-45d1-8cdd-6efe589ef917" containerName="maas-api"
Jul 27 22:16:03.102551 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.100567    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf13facf-f5b9-45d1-8cdd-6efe589ef917" containerName="maas-api"
Jul 27 22:16:03.102551 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.100735    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ef8d574-b1e0-4f4e-b0d5-e191b7d4a52d" containerName="payload-processing"
Jul 27 22:16:03.102551 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.100757    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb14c347-2128-45a8-92f7-b9a17fb5989e" containerName="payload-pre-processing"
Jul 27 22:16:03.102551 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.100776    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf13facf-f5b9-45d1-8cdd-6efe589ef917" containerName="maas-api"
Jul 27 22:16:03.105344 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.105315    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:03.109083 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.109044    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2671d368-serving-cert\""
Jul 27 22:16:03.124597 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.124560    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"]
Jul 27 22:16:03.177767 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.177710    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7h4m\" (UniqueName: \"kubernetes.io/projected/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-kube-api-access-z7h4m\") pod \"maas-api-e2e-mt-2671d368-7894ff779c-4w7mz\" (UID: \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:03.177767 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.177767    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-maas-api-tls\") pod \"maas-api-e2e-mt-2671d368-7894ff779c-4w7mz\" (UID: \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:03.178011 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.177803    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-openshift-service-ca\") pod \"maas-api-e2e-mt-2671d368-7894ff779c-4w7mz\" (UID: \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:03.278409 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.278368    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7h4m\" (UniqueName: \"kubernetes.io/projected/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-kube-api-access-z7h4m\") pod \"maas-api-e2e-mt-2671d368-7894ff779c-4w7mz\" (UID: \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:03.278629 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.278415    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-maas-api-tls\") pod \"maas-api-e2e-mt-2671d368-7894ff779c-4w7mz\" (UID: \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:03.278629 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:16:03.278562    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-serving-cert: secret "maas-api-e2e-mt-2671d368-serving-cert" not found
Jul 27 22:16:03.278629 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.278597    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-openshift-service-ca\") pod \"maas-api-e2e-mt-2671d368-7894ff779c-4w7mz\" (UID: \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:03.278876 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:16:03.278649    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-maas-api-tls podName:f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a nodeName:}" failed. No retries permitted until 2026-07-27 22:16:03.778624716 +0000 UTC m=+1848.589972067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-maas-api-tls") pod "maas-api-e2e-mt-2671d368-7894ff779c-4w7mz" (UID: "f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a") : secret "maas-api-e2e-mt-2671d368-serving-cert" not found
Jul 27 22:16:03.279460 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.279437    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-openshift-service-ca\") pod \"maas-api-e2e-mt-2671d368-7894ff779c-4w7mz\" (UID: \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:03.296650 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.296624    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7h4m\" (UniqueName: \"kubernetes.io/projected/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-kube-api-access-z7h4m\") pod \"maas-api-e2e-mt-2671d368-7894ff779c-4w7mz\" (UID: \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:03.785287 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:03.785242    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-maas-api-tls\") pod \"maas-api-e2e-mt-2671d368-7894ff779c-4w7mz\" (UID: \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:03.785515 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:16:03.785417    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-serving-cert: secret "maas-api-e2e-mt-2671d368-serving-cert" not found
Jul 27 22:16:03.785515 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:16:03.785493    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-maas-api-tls podName:f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a nodeName:}" failed. No retries permitted until 2026-07-27 22:16:04.785473158 +0000 UTC m=+1849.596820508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-maas-api-tls") pod "maas-api-e2e-mt-2671d368-7894ff779c-4w7mz" (UID: "f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a") : secret "maas-api-e2e-mt-2671d368-serving-cert" not found
Jul 27 22:16:04.241214 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.241176    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"]
Jul 27 22:16:04.244968 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.244945    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"
Jul 27 22:16:04.250202 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.250171    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2671d368-dockercfg-22kxh\""
Jul 27 22:16:04.250362 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.250257    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2671d368\""
Jul 27 22:16:04.266016 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.265988    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"]
Jul 27 22:16:04.290569 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.290524    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9v9z\" (UniqueName: \"kubernetes.io/projected/cac84b47-d4f4-4213-9dc5-d82990f35fc9-kube-api-access-v9v9z\") pod \"payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47\" (UID: \"cac84b47-d4f4-4213-9dc5-d82990f35fc9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"
Jul 27 22:16:04.290746 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.290689    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac84b47-d4f4-4213-9dc5-d82990f35fc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47\" (UID: \"cac84b47-d4f4-4213-9dc5-d82990f35fc9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"
Jul 27 22:16:04.320094 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.320055    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"]
Jul 27 22:16:04.323663 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.323642    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"
Jul 27 22:16:04.335681 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.335653    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"]
Jul 27 22:16:04.392032 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.391990    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb11b6f-7ac4-460a-b993-a9afd818c3d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6\" (UID: \"5bb11b6f-7ac4-460a-b993-a9afd818c3d2\") " pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"
Jul 27 22:16:04.392200 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.392075    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac84b47-d4f4-4213-9dc5-d82990f35fc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47\" (UID: \"cac84b47-d4f4-4213-9dc5-d82990f35fc9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"
Jul 27 22:16:04.392200 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.392119    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4mp\" (UniqueName: \"kubernetes.io/projected/5bb11b6f-7ac4-460a-b993-a9afd818c3d2-kube-api-access-rw4mp\") pod \"payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6\" (UID: \"5bb11b6f-7ac4-460a-b993-a9afd818c3d2\") " pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"
Jul 27 22:16:04.392200 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.392180    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9v9z\" (UniqueName: \"kubernetes.io/projected/cac84b47-d4f4-4213-9dc5-d82990f35fc9-kube-api-access-v9v9z\") pod \"payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47\" (UID: \"cac84b47-d4f4-4213-9dc5-d82990f35fc9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"
Jul 27 22:16:04.392835 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.392816    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac84b47-d4f4-4213-9dc5-d82990f35fc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47\" (UID: \"cac84b47-d4f4-4213-9dc5-d82990f35fc9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"
Jul 27 22:16:04.408721 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.408688    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9v9z\" (UniqueName: \"kubernetes.io/projected/cac84b47-d4f4-4213-9dc5-d82990f35fc9-kube-api-access-v9v9z\") pod \"payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47\" (UID: \"cac84b47-d4f4-4213-9dc5-d82990f35fc9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"
Jul 27 22:16:04.493581 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.493484    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb11b6f-7ac4-460a-b993-a9afd818c3d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6\" (UID: \"5bb11b6f-7ac4-460a-b993-a9afd818c3d2\") " pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"
Jul 27 22:16:04.493581 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.493556    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4mp\" (UniqueName: \"kubernetes.io/projected/5bb11b6f-7ac4-460a-b993-a9afd818c3d2-kube-api-access-rw4mp\") pod \"payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6\" (UID: \"5bb11b6f-7ac4-460a-b993-a9afd818c3d2\") " pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"
Jul 27 22:16:04.494316 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.494291    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb11b6f-7ac4-460a-b993-a9afd818c3d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6\" (UID: \"5bb11b6f-7ac4-460a-b993-a9afd818c3d2\") " pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"
Jul 27 22:16:04.507712 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.507676    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4mp\" (UniqueName: \"kubernetes.io/projected/5bb11b6f-7ac4-460a-b993-a9afd818c3d2-kube-api-access-rw4mp\") pod \"payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6\" (UID: \"5bb11b6f-7ac4-460a-b993-a9afd818c3d2\") " pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"
Jul 27 22:16:04.556357 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.556324    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"
Jul 27 22:16:04.634000 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.633968    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"
Jul 27 22:16:04.727536 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.727298    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"]
Jul 27 22:16:04.730734 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:16:04.730697    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac84b47_d4f4_4213_9dc5_d82990f35fc9.slice/crio-392df1b128435594f4a8629aae8fe04649056a8803b842b529d8e9ba209f0bfe WatchSource:0}: Error finding container 392df1b128435594f4a8629aae8fe04649056a8803b842b529d8e9ba209f0bfe: Status 404 returned error can't find the container with id 392df1b128435594f4a8629aae8fe04649056a8803b842b529d8e9ba209f0bfe
Jul 27 22:16:04.777577 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.777547    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47" event={"ID":"cac84b47-d4f4-4213-9dc5-d82990f35fc9","Type":"ContainerStarted","Data":"392df1b128435594f4a8629aae8fe04649056a8803b842b529d8e9ba209f0bfe"}
Jul 27 22:16:04.797515 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.797464    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-maas-api-tls\") pod \"maas-api-e2e-mt-2671d368-7894ff779c-4w7mz\" (UID: \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:04.800201 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.800178    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-maas-api-tls\") pod \"maas-api-e2e-mt-2671d368-7894ff779c-4w7mz\" (UID: \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:04.807383 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.807354    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"]
Jul 27 22:16:04.809817 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:16:04.809791    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb11b6f_7ac4_460a_b993_a9afd818c3d2.slice/crio-2eace81d350cca95c5233186d26fadafbea837f3b545d9dd30be2f61ec31bf83 WatchSource:0}: Error finding container 2eace81d350cca95c5233186d26fadafbea837f3b545d9dd30be2f61ec31bf83: Status 404 returned error can't find the container with id 2eace81d350cca95c5233186d26fadafbea837f3b545d9dd30be2f61ec31bf83
Jul 27 22:16:04.918777 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:04.918741    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:05.083363 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:05.083032    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"]
Jul 27 22:16:05.086920 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:16:05.086110    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45cfd9a_d3b4_41bc_b84f_2a6096b9d09a.slice/crio-82e82fd81d2682fb8d6cf79428d05d0f067369f9399753e2936e9fcf33e18afe WatchSource:0}: Error finding container 82e82fd81d2682fb8d6cf79428d05d0f067369f9399753e2936e9fcf33e18afe: Status 404 returned error can't find the container with id 82e82fd81d2682fb8d6cf79428d05d0f067369f9399753e2936e9fcf33e18afe
Jul 27 22:16:05.786734 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:05.786681    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz" event={"ID":"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a","Type":"ContainerStarted","Data":"2d9de0c8031fbd47afff9b74d2f96c31e9d9529f2c00578ea2a11e174f2b1a14"}
Jul 27 22:16:05.787243 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:05.786778    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz" event={"ID":"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a","Type":"ContainerStarted","Data":"82e82fd81d2682fb8d6cf79428d05d0f067369f9399753e2936e9fcf33e18afe"}
Jul 27 22:16:05.787530 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:05.787436    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:05.788892 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:05.788863    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6" event={"ID":"5bb11b6f-7ac4-460a-b993-a9afd818c3d2","Type":"ContainerStarted","Data":"814a901626c9e59a48baf128a1d483cd9d6a64cb2d381c6889aae82b93d353c6"}
Jul 27 22:16:05.789029 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:05.788949    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"
Jul 27 22:16:05.789101 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:05.789078    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6" event={"ID":"5bb11b6f-7ac4-460a-b993-a9afd818c3d2","Type":"ContainerStarted","Data":"2eace81d350cca95c5233186d26fadafbea837f3b545d9dd30be2f61ec31bf83"}
Jul 27 22:16:05.791020 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:05.790965    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47" event={"ID":"cac84b47-d4f4-4213-9dc5-d82990f35fc9","Type":"ContainerStarted","Data":"1dd890c63d434eb8f426fe942da364d0aacdc99e68de245f5f06127f172f45da"}
Jul 27 22:16:05.791260 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:05.791242    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"
Jul 27 22:16:05.864011 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:05.863956    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz" podStartSLOduration=2.296463207 podStartE2EDuration="2.863938551s" totalImagesPullingTime="567.475344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:16:05.82758528 +0000 UTC m=+1850.638932653" watchObservedRunningTime="2026-07-27 22:16:05.863938551 +0000 UTC m=+1850.675285922"
Jul 27 22:16:05.864214 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:05.864052    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6" podStartSLOduration=1.8640467950000001 podStartE2EDuration="1.864046795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:16:05.861590006 +0000 UTC m=+1850.672937378" watchObservedRunningTime="2026-07-27 22:16:05.864046795 +0000 UTC m=+1850.675394145"
Jul 27 22:16:05.893419 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:05.893348    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47" podStartSLOduration=1.893326297 podStartE2EDuration="1.893326297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:16:05.892014801 +0000 UTC m=+1850.703362205" watchObservedRunningTime="2026-07-27 22:16:05.893326297 +0000 UTC m=+1850.704673671"
Jul 27 22:16:12.811873 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:12.811839    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:16.803153 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:16.803116    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"
Jul 27 22:16:16.803633 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:16.803201    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"
Jul 27 22:16:24.830328 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:24.830290    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"]
Jul 27 22:16:24.830734 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:24.830539    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz" podUID="f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a" containerName="maas-api" containerID="cri-o://2d9de0c8031fbd47afff9b74d2f96c31e9d9529f2c00578ea2a11e174f2b1a14" gracePeriod=30
Jul 27 22:16:24.890150 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:24.890111    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"]
Jul 27 22:16:24.890672 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:24.890615    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6" podUID="5bb11b6f-7ac4-460a-b993-a9afd818c3d2" containerName="payload-processing" containerID="cri-o://814a901626c9e59a48baf128a1d483cd9d6a64cb2d381c6889aae82b93d353c6" gracePeriod=30
Jul 27 22:16:24.899272 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:24.896476    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"]
Jul 27 22:16:24.899272 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:24.896711    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47" podUID="cac84b47-d4f4-4213-9dc5-d82990f35fc9" containerName="payload-pre-processing" containerID="cri-o://1dd890c63d434eb8f426fe942da364d0aacdc99e68de245f5f06127f172f45da" gracePeriod=30
Jul 27 22:16:25.171282 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.171238    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:25.188165 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.188043    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"
Jul 27 22:16:25.195565 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.195535    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"
Jul 27 22:16:25.284668 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.284631    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7h4m\" (UniqueName: \"kubernetes.io/projected/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-kube-api-access-z7h4m\") pod \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\" (UID: \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\") "
Jul 27 22:16:25.284847 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.284696    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9v9z\" (UniqueName: \"kubernetes.io/projected/cac84b47-d4f4-4213-9dc5-d82990f35fc9-kube-api-access-v9v9z\") pod \"cac84b47-d4f4-4213-9dc5-d82990f35fc9\" (UID: \"cac84b47-d4f4-4213-9dc5-d82990f35fc9\") "
Jul 27 22:16:25.284847 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.284725    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac84b47-d4f4-4213-9dc5-d82990f35fc9-plugins-config-volume\") pod \"cac84b47-d4f4-4213-9dc5-d82990f35fc9\" (UID: \"cac84b47-d4f4-4213-9dc5-d82990f35fc9\") "
Jul 27 22:16:25.284979 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.284865    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-maas-api-tls\") pod \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\" (UID: \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\") "
Jul 27 22:16:25.284979 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.284950    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-openshift-service-ca\") pod \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\" (UID: \"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a\") "
Jul 27 22:16:25.285146 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.285121    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac84b47-d4f4-4213-9dc5-d82990f35fc9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cac84b47-d4f4-4213-9dc5-d82990f35fc9" (UID: "cac84b47-d4f4-4213-9dc5-d82990f35fc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:16:25.285440 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.285298    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac84b47-d4f4-4213-9dc5-d82990f35fc9-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:16:25.285659 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.285470    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a" (UID: "f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:16:25.287096 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.287065    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a" (UID: "f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:16:25.287191 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.287122    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-kube-api-access-z7h4m" (OuterVolumeSpecName: "kube-api-access-z7h4m") pod "f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a" (UID: "f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a"). InnerVolumeSpecName "kube-api-access-z7h4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:16:25.287191 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.287130    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac84b47-d4f4-4213-9dc5-d82990f35fc9-kube-api-access-v9v9z" (OuterVolumeSpecName: "kube-api-access-v9v9z") pod "cac84b47-d4f4-4213-9dc5-d82990f35fc9" (UID: "cac84b47-d4f4-4213-9dc5-d82990f35fc9"). InnerVolumeSpecName "kube-api-access-v9v9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:16:25.385992 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.385956    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw4mp\" (UniqueName: \"kubernetes.io/projected/5bb11b6f-7ac4-460a-b993-a9afd818c3d2-kube-api-access-rw4mp\") pod \"5bb11b6f-7ac4-460a-b993-a9afd818c3d2\" (UID: \"5bb11b6f-7ac4-460a-b993-a9afd818c3d2\") "
Jul 27 22:16:25.386171 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.386023    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb11b6f-7ac4-460a-b993-a9afd818c3d2-plugins-config-volume\") pod \"5bb11b6f-7ac4-460a-b993-a9afd818c3d2\" (UID: \"5bb11b6f-7ac4-460a-b993-a9afd818c3d2\") "
Jul 27 22:16:25.386291 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.386274    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:16:25.386346 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.386299    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7h4m\" (UniqueName: \"kubernetes.io/projected/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-kube-api-access-z7h4m\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:16:25.386346 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.386313    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9v9z\" (UniqueName: \"kubernetes.io/projected/cac84b47-d4f4-4213-9dc5-d82990f35fc9-kube-api-access-v9v9z\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:16:25.386346 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.386328    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:16:25.386462 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.386439    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb11b6f-7ac4-460a-b993-a9afd818c3d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5bb11b6f-7ac4-460a-b993-a9afd818c3d2" (UID: "5bb11b6f-7ac4-460a-b993-a9afd818c3d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:16:25.388332 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.388308    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb11b6f-7ac4-460a-b993-a9afd818c3d2-kube-api-access-rw4mp" (OuterVolumeSpecName: "kube-api-access-rw4mp") pod "5bb11b6f-7ac4-460a-b993-a9afd818c3d2" (UID: "5bb11b6f-7ac4-460a-b993-a9afd818c3d2"). InnerVolumeSpecName "kube-api-access-rw4mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:16:25.487740 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.487701    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb11b6f-7ac4-460a-b993-a9afd818c3d2-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:16:25.487740 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.487741    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw4mp\" (UniqueName: \"kubernetes.io/projected/5bb11b6f-7ac4-460a-b993-a9afd818c3d2-kube-api-access-rw4mp\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:16:25.899457 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.899423    2549 generic.go:356] "Generic (PLEG): container finished" podID="f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a" containerID="2d9de0c8031fbd47afff9b74d2f96c31e9d9529f2c00578ea2a11e174f2b1a14" exitCode=0
Jul 27 22:16:25.899948 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.899462    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"
Jul 27 22:16:25.899948 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.899504    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz" event={"ID":"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a","Type":"ContainerDied","Data":"2d9de0c8031fbd47afff9b74d2f96c31e9d9529f2c00578ea2a11e174f2b1a14"}
Jul 27 22:16:25.899948 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.899567    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz" event={"ID":"f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a","Type":"ContainerDied","Data":"82e82fd81d2682fb8d6cf79428d05d0f067369f9399753e2936e9fcf33e18afe"}
Jul 27 22:16:25.899948 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.899602    2549 scope.go:117] "RemoveContainer" containerID="2d9de0c8031fbd47afff9b74d2f96c31e9d9529f2c00578ea2a11e174f2b1a14"
Jul 27 22:16:25.901012 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.900916    2549 generic.go:356] "Generic (PLEG): container finished" podID="5bb11b6f-7ac4-460a-b993-a9afd818c3d2" containerID="814a901626c9e59a48baf128a1d483cd9d6a64cb2d381c6889aae82b93d353c6" exitCode=0
Jul 27 22:16:25.901012 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.900948    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"
Jul 27 22:16:25.901012 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.901000    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6" event={"ID":"5bb11b6f-7ac4-460a-b993-a9afd818c3d2","Type":"ContainerDied","Data":"814a901626c9e59a48baf128a1d483cd9d6a64cb2d381c6889aae82b93d353c6"}
Jul 27 22:16:25.901174 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.901047    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6" event={"ID":"5bb11b6f-7ac4-460a-b993-a9afd818c3d2","Type":"ContainerDied","Data":"2eace81d350cca95c5233186d26fadafbea837f3b545d9dd30be2f61ec31bf83"}
Jul 27 22:16:25.902635 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.902611    2549 generic.go:356] "Generic (PLEG): container finished" podID="cac84b47-d4f4-4213-9dc5-d82990f35fc9" containerID="1dd890c63d434eb8f426fe942da364d0aacdc99e68de245f5f06127f172f45da" exitCode=0
Jul 27 22:16:25.902753 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.902644    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"
Jul 27 22:16:25.902753 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.902693    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47" event={"ID":"cac84b47-d4f4-4213-9dc5-d82990f35fc9","Type":"ContainerDied","Data":"1dd890c63d434eb8f426fe942da364d0aacdc99e68de245f5f06127f172f45da"}
Jul 27 22:16:25.902857 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.902753    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47" event={"ID":"cac84b47-d4f4-4213-9dc5-d82990f35fc9","Type":"ContainerDied","Data":"392df1b128435594f4a8629aae8fe04649056a8803b842b529d8e9ba209f0bfe"}
Jul 27 22:16:25.909758 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.909741    2549 scope.go:117] "RemoveContainer" containerID="2d9de0c8031fbd47afff9b74d2f96c31e9d9529f2c00578ea2a11e174f2b1a14"
Jul 27 22:16:25.910068 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:16:25.910050    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9de0c8031fbd47afff9b74d2f96c31e9d9529f2c00578ea2a11e174f2b1a14\": container with ID starting with 2d9de0c8031fbd47afff9b74d2f96c31e9d9529f2c00578ea2a11e174f2b1a14 not found: ID does not exist" containerID="2d9de0c8031fbd47afff9b74d2f96c31e9d9529f2c00578ea2a11e174f2b1a14"
Jul 27 22:16:25.916264 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.910076    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9de0c8031fbd47afff9b74d2f96c31e9d9529f2c00578ea2a11e174f2b1a14"} err="failed to get container status \"2d9de0c8031fbd47afff9b74d2f96c31e9d9529f2c00578ea2a11e174f2b1a14\": rpc error: code = NotFound desc = could not find container \"2d9de0c8031fbd47afff9b74d2f96c31e9d9529f2c00578ea2a11e174f2b1a14\": container with ID starting with 2d9de0c8031fbd47afff9b74d2f96c31e9d9529f2c00578ea2a11e174f2b1a14 not found: ID does not exist"
Jul 27 22:16:25.916264 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.910428    2549 scope.go:117] "RemoveContainer" containerID="814a901626c9e59a48baf128a1d483cd9d6a64cb2d381c6889aae82b93d353c6"
Jul 27 22:16:25.924997 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.924972    2549 scope.go:117] "RemoveContainer" containerID="814a901626c9e59a48baf128a1d483cd9d6a64cb2d381c6889aae82b93d353c6"
Jul 27 22:16:25.925280 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:16:25.925258    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"814a901626c9e59a48baf128a1d483cd9d6a64cb2d381c6889aae82b93d353c6\": container with ID starting with 814a901626c9e59a48baf128a1d483cd9d6a64cb2d381c6889aae82b93d353c6 not found: ID does not exist" containerID="814a901626c9e59a48baf128a1d483cd9d6a64cb2d381c6889aae82b93d353c6"
Jul 27 22:16:25.925341 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.925289    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"814a901626c9e59a48baf128a1d483cd9d6a64cb2d381c6889aae82b93d353c6"} err="failed to get container status \"814a901626c9e59a48baf128a1d483cd9d6a64cb2d381c6889aae82b93d353c6\": rpc error: code = NotFound desc = could not find container \"814a901626c9e59a48baf128a1d483cd9d6a64cb2d381c6889aae82b93d353c6\": container with ID starting with 814a901626c9e59a48baf128a1d483cd9d6a64cb2d381c6889aae82b93d353c6 not found: ID does not exist"
Jul 27 22:16:25.925341 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.925306    2549 scope.go:117] "RemoveContainer" containerID="1dd890c63d434eb8f426fe942da364d0aacdc99e68de245f5f06127f172f45da"
Jul 27 22:16:25.931525 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.931501    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"]
Jul 27 22:16:25.933246 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.933206    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2671d368-6986bc968b-k6lc6"]
Jul 27 22:16:25.934277 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.934251    2549 scope.go:117] "RemoveContainer" containerID="1dd890c63d434eb8f426fe942da364d0aacdc99e68de245f5f06127f172f45da"
Jul 27 22:16:25.934621 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:16:25.934600    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd890c63d434eb8f426fe942da364d0aacdc99e68de245f5f06127f172f45da\": container with ID starting with 1dd890c63d434eb8f426fe942da364d0aacdc99e68de245f5f06127f172f45da not found: ID does not exist" containerID="1dd890c63d434eb8f426fe942da364d0aacdc99e68de245f5f06127f172f45da"
Jul 27 22:16:25.934681 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.934632    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd890c63d434eb8f426fe942da364d0aacdc99e68de245f5f06127f172f45da"} err="failed to get container status \"1dd890c63d434eb8f426fe942da364d0aacdc99e68de245f5f06127f172f45da\": rpc error: code = NotFound desc = could not find container \"1dd890c63d434eb8f426fe942da364d0aacdc99e68de245f5f06127f172f45da\": container with ID starting with 1dd890c63d434eb8f426fe942da364d0aacdc99e68de245f5f06127f172f45da not found: ID does not exist"
Jul 27 22:16:25.944890 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.944858    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"]
Jul 27 22:16:25.951988 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.951955    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2671d368-7894ff779c-4w7mz"]
Jul 27 22:16:25.964483 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.964411    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"]
Jul 27 22:16:25.966070 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:25.966049    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2671d368-849bccfdd8-qcv47"]
Jul 27 22:16:27.700632 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:27.700594    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bb11b6f-7ac4-460a-b993-a9afd818c3d2" path="/var/lib/kubelet/pods/5bb11b6f-7ac4-460a-b993-a9afd818c3d2/volumes"
Jul 27 22:16:27.701023 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:27.700943    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cac84b47-d4f4-4213-9dc5-d82990f35fc9" path="/var/lib/kubelet/pods/cac84b47-d4f4-4213-9dc5-d82990f35fc9/volumes"
Jul 27 22:16:27.701304 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:27.701290    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a" path="/var/lib/kubelet/pods/f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a/volumes"
Jul 27 22:16:59.369305 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.369206    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"]
Jul 27 22:16:59.369757 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.369634    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb11b6f-7ac4-460a-b993-a9afd818c3d2" containerName="payload-processing"
Jul 27 22:16:59.369757 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.369646    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb11b6f-7ac4-460a-b993-a9afd818c3d2" containerName="payload-processing"
Jul 27 22:16:59.369757 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.369657    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a" containerName="maas-api"
Jul 27 22:16:59.369757 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.369663    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a" containerName="maas-api"
Jul 27 22:16:59.369757 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.369674    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac84b47-d4f4-4213-9dc5-d82990f35fc9" containerName="payload-pre-processing"
Jul 27 22:16:59.369757 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.369679    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac84b47-d4f4-4213-9dc5-d82990f35fc9" containerName="payload-pre-processing"
Jul 27 22:16:59.369757 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.369740    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb11b6f-7ac4-460a-b993-a9afd818c3d2" containerName="payload-processing"
Jul 27 22:16:59.369757 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.369751    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="f45cfd9a-d3b4-41bc-b84f-2a6096b9d09a" containerName="maas-api"
Jul 27 22:16:59.369757 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.369757    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac84b47-d4f4-4213-9dc5-d82990f35fc9" containerName="payload-pre-processing"
Jul 27 22:16:59.372455 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.372438    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:16:59.375673 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.375646    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5d6c9eb6-serving-cert\""
Jul 27 22:16:59.386160 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.386131    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"]
Jul 27 22:16:59.493852 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.493808    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/983b7d84-81a7-4045-93dc-5b3baa268eb2-openshift-service-ca\") pod \"maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g\" (UID: \"983b7d84-81a7-4045-93dc-5b3baa268eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:16:59.494044 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.493892    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4n56\" (UniqueName: \"kubernetes.io/projected/983b7d84-81a7-4045-93dc-5b3baa268eb2-kube-api-access-v4n56\") pod \"maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g\" (UID: \"983b7d84-81a7-4045-93dc-5b3baa268eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:16:59.494044 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.493931    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/983b7d84-81a7-4045-93dc-5b3baa268eb2-maas-api-tls\") pod \"maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g\" (UID: \"983b7d84-81a7-4045-93dc-5b3baa268eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:16:59.594623 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.594587    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/983b7d84-81a7-4045-93dc-5b3baa268eb2-openshift-service-ca\") pod \"maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g\" (UID: \"983b7d84-81a7-4045-93dc-5b3baa268eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:16:59.594830 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.594637    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4n56\" (UniqueName: \"kubernetes.io/projected/983b7d84-81a7-4045-93dc-5b3baa268eb2-kube-api-access-v4n56\") pod \"maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g\" (UID: \"983b7d84-81a7-4045-93dc-5b3baa268eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:16:59.594830 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.594684    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/983b7d84-81a7-4045-93dc-5b3baa268eb2-maas-api-tls\") pod \"maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g\" (UID: \"983b7d84-81a7-4045-93dc-5b3baa268eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:16:59.595943 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.595902    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/983b7d84-81a7-4045-93dc-5b3baa268eb2-openshift-service-ca\") pod \"maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g\" (UID: \"983b7d84-81a7-4045-93dc-5b3baa268eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:16:59.602731 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.602701    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/983b7d84-81a7-4045-93dc-5b3baa268eb2-maas-api-tls\") pod \"maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g\" (UID: \"983b7d84-81a7-4045-93dc-5b3baa268eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:16:59.604882 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.604862    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4n56\" (UniqueName: \"kubernetes.io/projected/983b7d84-81a7-4045-93dc-5b3baa268eb2-kube-api-access-v4n56\") pod \"maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g\" (UID: \"983b7d84-81a7-4045-93dc-5b3baa268eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:16:59.683283 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.683156    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:16:59.845299 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.845263    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"]
Jul 27 22:16:59.846461 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:16:59.846430    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983b7d84_81a7_4045_93dc_5b3baa268eb2.slice/crio-bd034877a1ea9c30f230ff79377eb21ae024400b086d40a36bc984014fb81f13 WatchSource:0}: Error finding container bd034877a1ea9c30f230ff79377eb21ae024400b086d40a36bc984014fb81f13: Status 404 returned error can't find the container with id bd034877a1ea9c30f230ff79377eb21ae024400b086d40a36bc984014fb81f13
Jul 27 22:16:59.847985 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:16:59.847967    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 22:17:00.074009 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.073912    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g" event={"ID":"983b7d84-81a7-4045-93dc-5b3baa268eb2","Type":"ContainerStarted","Data":"bd034877a1ea9c30f230ff79377eb21ae024400b086d40a36bc984014fb81f13"}
Jul 27 22:17:00.217947 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.217910    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"]
Jul 27 22:17:00.221344 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.221325    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"
Jul 27 22:17:00.224179 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.224155    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5d6c9eb6\""
Jul 27 22:17:00.231390 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.231363    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"]
Jul 27 22:17:00.257806 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.257780    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"]
Jul 27 22:17:00.261458 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.261438    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"
Jul 27 22:17:00.264675 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.264646    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5d6c9eb6-dockercfg-56tch\""
Jul 27 22:17:00.273783 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.273750    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"]
Jul 27 22:17:00.301418 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.301391    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea08a88e-7fef-420d-84f5-4fcef5474169-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9\" (UID: \"ea08a88e-7fef-420d-84f5-4fcef5474169\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"
Jul 27 22:17:00.301541 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.301521    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2bgp\" (UniqueName: \"kubernetes.io/projected/ea08a88e-7fef-420d-84f5-4fcef5474169-kube-api-access-v2bgp\") pod \"payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9\" (UID: \"ea08a88e-7fef-420d-84f5-4fcef5474169\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"
Jul 27 22:17:00.402216 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.402173    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2bgp\" (UniqueName: \"kubernetes.io/projected/ea08a88e-7fef-420d-84f5-4fcef5474169-kube-api-access-v2bgp\") pod \"payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9\" (UID: \"ea08a88e-7fef-420d-84f5-4fcef5474169\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"
Jul 27 22:17:00.402651 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.402281    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c875af00-cb60-4547-8472-e9454812fd5b-plugins-config-volume\") pod \"payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587\" (UID: \"c875af00-cb60-4547-8472-e9454812fd5b\") " pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"
Jul 27 22:17:00.402651 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.402350    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rpln\" (UniqueName: \"kubernetes.io/projected/c875af00-cb60-4547-8472-e9454812fd5b-kube-api-access-5rpln\") pod \"payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587\" (UID: \"c875af00-cb60-4547-8472-e9454812fd5b\") " pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"
Jul 27 22:17:00.402651 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.402395    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea08a88e-7fef-420d-84f5-4fcef5474169-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9\" (UID: \"ea08a88e-7fef-420d-84f5-4fcef5474169\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"
Jul 27 22:17:00.403034 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.403011    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea08a88e-7fef-420d-84f5-4fcef5474169-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9\" (UID: \"ea08a88e-7fef-420d-84f5-4fcef5474169\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"
Jul 27 22:17:00.412252 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.412188    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2bgp\" (UniqueName: \"kubernetes.io/projected/ea08a88e-7fef-420d-84f5-4fcef5474169-kube-api-access-v2bgp\") pod \"payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9\" (UID: \"ea08a88e-7fef-420d-84f5-4fcef5474169\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"
Jul 27 22:17:00.503683 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.503646    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c875af00-cb60-4547-8472-e9454812fd5b-plugins-config-volume\") pod \"payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587\" (UID: \"c875af00-cb60-4547-8472-e9454812fd5b\") " pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"
Jul 27 22:17:00.503879 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.503708    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rpln\" (UniqueName: \"kubernetes.io/projected/c875af00-cb60-4547-8472-e9454812fd5b-kube-api-access-5rpln\") pod \"payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587\" (UID: \"c875af00-cb60-4547-8472-e9454812fd5b\") " pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"
Jul 27 22:17:00.504381 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.504359    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c875af00-cb60-4547-8472-e9454812fd5b-plugins-config-volume\") pod \"payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587\" (UID: \"c875af00-cb60-4547-8472-e9454812fd5b\") " pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"
Jul 27 22:17:00.511777 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.511753    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rpln\" (UniqueName: \"kubernetes.io/projected/c875af00-cb60-4547-8472-e9454812fd5b-kube-api-access-5rpln\") pod \"payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587\" (UID: \"c875af00-cb60-4547-8472-e9454812fd5b\") " pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"
Jul 27 22:17:00.533786 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.533752    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"
Jul 27 22:17:00.591326 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.591215    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"
Jul 27 22:17:00.693629 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.693601    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"]
Jul 27 22:17:00.696212 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:17:00.696183    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea08a88e_7fef_420d_84f5_4fcef5474169.slice/crio-c00d618778b7ade758ed676c37bf8d1b4968cb9f2b62e1e53bea28c2ffa43ca9 WatchSource:0}: Error finding container c00d618778b7ade758ed676c37bf8d1b4968cb9f2b62e1e53bea28c2ffa43ca9: Status 404 returned error can't find the container with id c00d618778b7ade758ed676c37bf8d1b4968cb9f2b62e1e53bea28c2ffa43ca9
Jul 27 22:17:00.762968 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:00.762924    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"]
Jul 27 22:17:00.765610 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:17:00.765567    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc875af00_cb60_4547_8472_e9454812fd5b.slice/crio-0c8308e85379c22889b908deeed991293bc6af24b5c1d265e8fffc613ffcded5 WatchSource:0}: Error finding container 0c8308e85379c22889b908deeed991293bc6af24b5c1d265e8fffc613ffcded5: Status 404 returned error can't find the container with id 0c8308e85379c22889b908deeed991293bc6af24b5c1d265e8fffc613ffcded5
Jul 27 22:17:01.081359 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:01.081264    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g" event={"ID":"983b7d84-81a7-4045-93dc-5b3baa268eb2","Type":"ContainerStarted","Data":"b4709f9d226b0153eb0079984b14f824b7256bb6a9384a98b571580f467332a9"}
Jul 27 22:17:01.081359 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:01.081353    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:17:01.083046 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:01.083016    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587" event={"ID":"c875af00-cb60-4547-8472-e9454812fd5b","Type":"ContainerStarted","Data":"9ee625a12852fd0fdc921a24848cdd780d5e391328efc86ff27c8420b75c0beb"}
Jul 27 22:17:01.083212 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:01.083070    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587" event={"ID":"c875af00-cb60-4547-8472-e9454812fd5b","Type":"ContainerStarted","Data":"0c8308e85379c22889b908deeed991293bc6af24b5c1d265e8fffc613ffcded5"}
Jul 27 22:17:01.083212 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:01.083128    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"
Jul 27 22:17:01.085742 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:01.085502    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9" event={"ID":"ea08a88e-7fef-420d-84f5-4fcef5474169","Type":"ContainerStarted","Data":"069cd766aafdb32ff8a5c8c6d7c0d5eedf00dc5bbdac380c5c0f615aef5b3d54"}
Jul 27 22:17:01.085742 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:01.085571    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9" event={"ID":"ea08a88e-7fef-420d-84f5-4fcef5474169","Type":"ContainerStarted","Data":"c00d618778b7ade758ed676c37bf8d1b4968cb9f2b62e1e53bea28c2ffa43ca9"}
Jul 27 22:17:01.085942 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:01.085806    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"
Jul 27 22:17:01.101922 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:01.101872    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g" podStartSLOduration=1.712638079 podStartE2EDuration="2.101858332s" totalImagesPullingTime="389.220253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:16:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:17:01.099711036 +0000 UTC m=+1905.911058422" watchObservedRunningTime="2026-07-27 22:17:01.101858332 +0000 UTC m=+1905.913205704"
Jul 27 22:17:01.117794 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:01.117745    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9" podStartSLOduration=1.117730886 podStartE2EDuration="1.117730886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:17:01.115925308 +0000 UTC m=+1905.927272682" watchObservedRunningTime="2026-07-27 22:17:01.117730886 +0000 UTC m=+1905.929078258"
Jul 27 22:17:01.134785 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:01.134731    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587" podStartSLOduration=1.13471787 podStartE2EDuration="1.13471787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:17:01.132656973 +0000 UTC m=+1905.944004346" watchObservedRunningTime="2026-07-27 22:17:01.13471787 +0000 UTC m=+1905.946065241"
Jul 27 22:17:07.105167 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:07.105131    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:17:12.100547 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:12.100513    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"
Jul 27 22:17:12.101054 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:17:12.100597    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"
Jul 27 22:18:19.272637 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.271997    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"]
Jul 27 22:18:19.272637 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.272280    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g" podUID="983b7d84-81a7-4045-93dc-5b3baa268eb2" containerName="maas-api" containerID="cri-o://b4709f9d226b0153eb0079984b14f824b7256bb6a9384a98b571580f467332a9" gracePeriod=30
Jul 27 22:18:19.333415 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.333378    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"]
Jul 27 22:18:19.333681 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.333654    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587" podUID="c875af00-cb60-4547-8472-e9454812fd5b" containerName="payload-processing" containerID="cri-o://9ee625a12852fd0fdc921a24848cdd780d5e391328efc86ff27c8420b75c0beb" gracePeriod=30
Jul 27 22:18:19.347104 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.347005    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"]
Jul 27 22:18:19.488759 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.488725    2549 generic.go:356] "Generic (PLEG): container finished" podID="983b7d84-81a7-4045-93dc-5b3baa268eb2" containerID="b4709f9d226b0153eb0079984b14f824b7256bb6a9384a98b571580f467332a9" exitCode=0
Jul 27 22:18:19.488954 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.488806    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g" event={"ID":"983b7d84-81a7-4045-93dc-5b3baa268eb2","Type":"ContainerDied","Data":"b4709f9d226b0153eb0079984b14f824b7256bb6a9384a98b571580f467332a9"}
Jul 27 22:18:19.490660 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.490629    2549 generic.go:356] "Generic (PLEG): container finished" podID="c875af00-cb60-4547-8472-e9454812fd5b" containerID="9ee625a12852fd0fdc921a24848cdd780d5e391328efc86ff27c8420b75c0beb" exitCode=0
Jul 27 22:18:19.490921 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.490802    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587" event={"ID":"c875af00-cb60-4547-8472-e9454812fd5b","Type":"ContainerDied","Data":"9ee625a12852fd0fdc921a24848cdd780d5e391328efc86ff27c8420b75c0beb"}
Jul 27 22:18:19.491014 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.490945    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9" podUID="ea08a88e-7fef-420d-84f5-4fcef5474169" containerName="payload-pre-processing" containerID="cri-o://069cd766aafdb32ff8a5c8c6d7c0d5eedf00dc5bbdac380c5c0f615aef5b3d54" gracePeriod=30
Jul 27 22:18:19.614112 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.614089    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:18:19.617831 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.617803    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"
Jul 27 22:18:19.687942 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.687905    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c875af00-cb60-4547-8472-e9454812fd5b-plugins-config-volume\") pod \"c875af00-cb60-4547-8472-e9454812fd5b\" (UID: \"c875af00-cb60-4547-8472-e9454812fd5b\") "
Jul 27 22:18:19.688113 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.687961    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/983b7d84-81a7-4045-93dc-5b3baa268eb2-openshift-service-ca\") pod \"983b7d84-81a7-4045-93dc-5b3baa268eb2\" (UID: \"983b7d84-81a7-4045-93dc-5b3baa268eb2\") "
Jul 27 22:18:19.688113 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.688017    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4n56\" (UniqueName: \"kubernetes.io/projected/983b7d84-81a7-4045-93dc-5b3baa268eb2-kube-api-access-v4n56\") pod \"983b7d84-81a7-4045-93dc-5b3baa268eb2\" (UID: \"983b7d84-81a7-4045-93dc-5b3baa268eb2\") "
Jul 27 22:18:19.688113 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.688063    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rpln\" (UniqueName: \"kubernetes.io/projected/c875af00-cb60-4547-8472-e9454812fd5b-kube-api-access-5rpln\") pod \"c875af00-cb60-4547-8472-e9454812fd5b\" (UID: \"c875af00-cb60-4547-8472-e9454812fd5b\") "
Jul 27 22:18:19.688311 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.688134    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/983b7d84-81a7-4045-93dc-5b3baa268eb2-maas-api-tls\") pod \"983b7d84-81a7-4045-93dc-5b3baa268eb2\" (UID: \"983b7d84-81a7-4045-93dc-5b3baa268eb2\") "
Jul 27 22:18:19.688730 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.688366    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c875af00-cb60-4547-8472-e9454812fd5b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c875af00-cb60-4547-8472-e9454812fd5b" (UID: "c875af00-cb60-4547-8472-e9454812fd5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:18:19.688730 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.688454    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c875af00-cb60-4547-8472-e9454812fd5b-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:18:19.688895 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.688805    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983b7d84-81a7-4045-93dc-5b3baa268eb2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "983b7d84-81a7-4045-93dc-5b3baa268eb2" (UID: "983b7d84-81a7-4045-93dc-5b3baa268eb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:18:19.690720 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.690668    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983b7d84-81a7-4045-93dc-5b3baa268eb2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "983b7d84-81a7-4045-93dc-5b3baa268eb2" (UID: "983b7d84-81a7-4045-93dc-5b3baa268eb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:18:19.690844 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.690801    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c875af00-cb60-4547-8472-e9454812fd5b-kube-api-access-5rpln" (OuterVolumeSpecName: "kube-api-access-5rpln") pod "c875af00-cb60-4547-8472-e9454812fd5b" (UID: "c875af00-cb60-4547-8472-e9454812fd5b"). InnerVolumeSpecName "kube-api-access-5rpln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:18:19.690918 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.690880    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983b7d84-81a7-4045-93dc-5b3baa268eb2-kube-api-access-v4n56" (OuterVolumeSpecName: "kube-api-access-v4n56") pod "983b7d84-81a7-4045-93dc-5b3baa268eb2" (UID: "983b7d84-81a7-4045-93dc-5b3baa268eb2"). InnerVolumeSpecName "kube-api-access-v4n56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:18:19.761094 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.761068    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"
Jul 27 22:18:19.789435 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.789403    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea08a88e-7fef-420d-84f5-4fcef5474169-plugins-config-volume\") pod \"ea08a88e-7fef-420d-84f5-4fcef5474169\" (UID: \"ea08a88e-7fef-420d-84f5-4fcef5474169\") "
Jul 27 22:18:19.789580 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.789507    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2bgp\" (UniqueName: \"kubernetes.io/projected/ea08a88e-7fef-420d-84f5-4fcef5474169-kube-api-access-v2bgp\") pod \"ea08a88e-7fef-420d-84f5-4fcef5474169\" (UID: \"ea08a88e-7fef-420d-84f5-4fcef5474169\") "
Jul 27 22:18:19.789827 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.789806    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4n56\" (UniqueName: \"kubernetes.io/projected/983b7d84-81a7-4045-93dc-5b3baa268eb2-kube-api-access-v4n56\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:18:19.789827 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.789817    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea08a88e-7fef-420d-84f5-4fcef5474169-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea08a88e-7fef-420d-84f5-4fcef5474169" (UID: "ea08a88e-7fef-420d-84f5-4fcef5474169"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:18:19.789998 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.789832    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rpln\" (UniqueName: \"kubernetes.io/projected/c875af00-cb60-4547-8472-e9454812fd5b-kube-api-access-5rpln\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:18:19.789998 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.789845    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/983b7d84-81a7-4045-93dc-5b3baa268eb2-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:18:19.789998 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.789858    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/983b7d84-81a7-4045-93dc-5b3baa268eb2-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:18:19.791762 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.791741    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea08a88e-7fef-420d-84f5-4fcef5474169-kube-api-access-v2bgp" (OuterVolumeSpecName: "kube-api-access-v2bgp") pod "ea08a88e-7fef-420d-84f5-4fcef5474169" (UID: "ea08a88e-7fef-420d-84f5-4fcef5474169"). InnerVolumeSpecName "kube-api-access-v2bgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:18:19.890752 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.890710    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2bgp\" (UniqueName: \"kubernetes.io/projected/ea08a88e-7fef-420d-84f5-4fcef5474169-kube-api-access-v2bgp\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:18:19.890752 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:19.890744    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea08a88e-7fef-420d-84f5-4fcef5474169-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:18:20.497453 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.497416    2549 generic.go:356] "Generic (PLEG): container finished" podID="ea08a88e-7fef-420d-84f5-4fcef5474169" containerID="069cd766aafdb32ff8a5c8c6d7c0d5eedf00dc5bbdac380c5c0f615aef5b3d54" exitCode=0
Jul 27 22:18:20.497901 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.497499    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"
Jul 27 22:18:20.497901 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.497539    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9" event={"ID":"ea08a88e-7fef-420d-84f5-4fcef5474169","Type":"ContainerDied","Data":"069cd766aafdb32ff8a5c8c6d7c0d5eedf00dc5bbdac380c5c0f615aef5b3d54"}
Jul 27 22:18:20.497901 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.497604    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9" event={"ID":"ea08a88e-7fef-420d-84f5-4fcef5474169","Type":"ContainerDied","Data":"c00d618778b7ade758ed676c37bf8d1b4968cb9f2b62e1e53bea28c2ffa43ca9"}
Jul 27 22:18:20.497901 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.497633    2549 scope.go:117] "RemoveContainer" containerID="069cd766aafdb32ff8a5c8c6d7c0d5eedf00dc5bbdac380c5c0f615aef5b3d54"
Jul 27 22:18:20.506469 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.506313    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g" event={"ID":"983b7d84-81a7-4045-93dc-5b3baa268eb2","Type":"ContainerDied","Data":"bd034877a1ea9c30f230ff79377eb21ae024400b086d40a36bc984014fb81f13"}
Jul 27 22:18:20.506469 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.506386    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"
Jul 27 22:18:20.509058 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.509024    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587" event={"ID":"c875af00-cb60-4547-8472-e9454812fd5b","Type":"ContainerDied","Data":"0c8308e85379c22889b908deeed991293bc6af24b5c1d265e8fffc613ffcded5"}
Jul 27 22:18:20.509205 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.509107    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"
Jul 27 22:18:20.516392 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.516365    2549 scope.go:117] "RemoveContainer" containerID="069cd766aafdb32ff8a5c8c6d7c0d5eedf00dc5bbdac380c5c0f615aef5b3d54"
Jul 27 22:18:20.516685 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:18:20.516666    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"069cd766aafdb32ff8a5c8c6d7c0d5eedf00dc5bbdac380c5c0f615aef5b3d54\": container with ID starting with 069cd766aafdb32ff8a5c8c6d7c0d5eedf00dc5bbdac380c5c0f615aef5b3d54 not found: ID does not exist" containerID="069cd766aafdb32ff8a5c8c6d7c0d5eedf00dc5bbdac380c5c0f615aef5b3d54"
Jul 27 22:18:20.516772 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.516698    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"069cd766aafdb32ff8a5c8c6d7c0d5eedf00dc5bbdac380c5c0f615aef5b3d54"} err="failed to get container status \"069cd766aafdb32ff8a5c8c6d7c0d5eedf00dc5bbdac380c5c0f615aef5b3d54\": rpc error: code = NotFound desc = could not find container \"069cd766aafdb32ff8a5c8c6d7c0d5eedf00dc5bbdac380c5c0f615aef5b3d54\": container with ID starting with 069cd766aafdb32ff8a5c8c6d7c0d5eedf00dc5bbdac380c5c0f615aef5b3d54 not found: ID does not exist"
Jul 27 22:18:20.516772 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.516718    2549 scope.go:117] "RemoveContainer" containerID="b4709f9d226b0153eb0079984b14f824b7256bb6a9384a98b571580f467332a9"
Jul 27 22:18:20.525382 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.525315    2549 scope.go:117] "RemoveContainer" containerID="9ee625a12852fd0fdc921a24848cdd780d5e391328efc86ff27c8420b75c0beb"
Jul 27 22:18:20.528091 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.528068    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"]
Jul 27 22:18:20.532366 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.532342    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5d6c9eb6-57f8b5cb6-zsj4g"]
Jul 27 22:18:20.544340 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.544271    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"]
Jul 27 22:18:20.548350 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.548326    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5d6c9eb6-c4bbdfdb4-qn4x9"]
Jul 27 22:18:20.560503 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.560477    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"]
Jul 27 22:18:20.565056 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:20.565031    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5d6c9eb6-6cf85c6566-bg587"]
Jul 27 22:18:21.701419 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:21.701381    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="983b7d84-81a7-4045-93dc-5b3baa268eb2" path="/var/lib/kubelet/pods/983b7d84-81a7-4045-93dc-5b3baa268eb2/volumes"
Jul 27 22:18:21.701778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:21.701737    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c875af00-cb60-4547-8472-e9454812fd5b" path="/var/lib/kubelet/pods/c875af00-cb60-4547-8472-e9454812fd5b/volumes"
Jul 27 22:18:21.702038 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:18:21.702026    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea08a88e-7fef-420d-84f5-4fcef5474169" path="/var/lib/kubelet/pods/ea08a88e-7fef-420d-84f5-4fcef5474169/volumes"
Jul 27 22:20:14.268319 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.268286    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"]
Jul 27 22:20:14.268778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.268660    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983b7d84-81a7-4045-93dc-5b3baa268eb2" containerName="maas-api"
Jul 27 22:20:14.268778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.268671    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="983b7d84-81a7-4045-93dc-5b3baa268eb2" containerName="maas-api"
Jul 27 22:20:14.268778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.268681    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c875af00-cb60-4547-8472-e9454812fd5b" containerName="payload-processing"
Jul 27 22:20:14.268778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.268686    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c875af00-cb60-4547-8472-e9454812fd5b" containerName="payload-processing"
Jul 27 22:20:14.268778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.268705    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea08a88e-7fef-420d-84f5-4fcef5474169" containerName="payload-pre-processing"
Jul 27 22:20:14.268778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.268711    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea08a88e-7fef-420d-84f5-4fcef5474169" containerName="payload-pre-processing"
Jul 27 22:20:14.268778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.268768    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="983b7d84-81a7-4045-93dc-5b3baa268eb2" containerName="maas-api"
Jul 27 22:20:14.268778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.268777    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea08a88e-7fef-420d-84f5-4fcef5474169" containerName="payload-pre-processing"
Jul 27 22:20:14.269048 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.268788    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="c875af00-cb60-4547-8472-e9454812fd5b" containerName="payload-processing"
Jul 27 22:20:14.271449 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.271431    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:20:14.275542 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.275516    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7a6dea-serving-cert\""
Jul 27 22:20:14.283076 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.283031    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"]
Jul 27 22:20:14.414280 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.414244    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc4xp\" (UniqueName: \"kubernetes.io/projected/53ee2673-1eba-4089-a189-7b28c113f988-kube-api-access-jc4xp\") pod \"maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7\" (UID: \"53ee2673-1eba-4089-a189-7b28c113f988\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:20:14.414463 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.414320    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ee2673-1eba-4089-a189-7b28c113f988-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7\" (UID: \"53ee2673-1eba-4089-a189-7b28c113f988\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:20:14.414463 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.414342    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ee2673-1eba-4089-a189-7b28c113f988-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7\" (UID: \"53ee2673-1eba-4089-a189-7b28c113f988\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:20:14.515689 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.515641    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc4xp\" (UniqueName: \"kubernetes.io/projected/53ee2673-1eba-4089-a189-7b28c113f988-kube-api-access-jc4xp\") pod \"maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7\" (UID: \"53ee2673-1eba-4089-a189-7b28c113f988\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:20:14.515874 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.515728    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ee2673-1eba-4089-a189-7b28c113f988-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7\" (UID: \"53ee2673-1eba-4089-a189-7b28c113f988\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:20:14.515874 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.515758    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ee2673-1eba-4089-a189-7b28c113f988-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7\" (UID: \"53ee2673-1eba-4089-a189-7b28c113f988\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:20:14.515960 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:20:14.515867    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-serving-cert: secret "maas-api-e2e-inf-a-7a6dea-serving-cert" not found
Jul 27 22:20:14.515960 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:20:14.515938    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ee2673-1eba-4089-a189-7b28c113f988-maas-api-tls podName:53ee2673-1eba-4089-a189-7b28c113f988 nodeName:}" failed. No retries permitted until 2026-07-27 22:20:15.015918898 +0000 UTC m=+2099.827266257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53ee2673-1eba-4089-a189-7b28c113f988-maas-api-tls") pod "maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7" (UID: "53ee2673-1eba-4089-a189-7b28c113f988") : secret "maas-api-e2e-inf-a-7a6dea-serving-cert" not found
Jul 27 22:20:14.516353 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.516336    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ee2673-1eba-4089-a189-7b28c113f988-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7\" (UID: \"53ee2673-1eba-4089-a189-7b28c113f988\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:20:14.526842 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.526778    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc4xp\" (UniqueName: \"kubernetes.io/projected/53ee2673-1eba-4089-a189-7b28c113f988-kube-api-access-jc4xp\") pod \"maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7\" (UID: \"53ee2673-1eba-4089-a189-7b28c113f988\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:20:14.987973 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.987933    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"]
Jul 27 22:20:14.991603 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.991580    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"
Jul 27 22:20:14.994886 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.994860    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7a6dea-dockercfg-kbbw5\""
Jul 27 22:20:14.994886 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:14.994883    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7a6dea\""
Jul 27 22:20:15.003657 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.003630    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"]
Jul 27 22:20:15.020607 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.020571    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ee2673-1eba-4089-a189-7b28c113f988-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7\" (UID: \"53ee2673-1eba-4089-a189-7b28c113f988\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:20:15.023342 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.023318    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ee2673-1eba-4089-a189-7b28c113f988-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7\" (UID: \"53ee2673-1eba-4089-a189-7b28c113f988\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:20:15.026548 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.026523    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"]
Jul 27 22:20:15.029995 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.029975    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"
Jul 27 22:20:15.040497 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.040475    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"]
Jul 27 22:20:15.121694 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.121653    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da4ecd8-addc-4549-8e73-6a6d1f60a71e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx\" (UID: \"5da4ecd8-addc-4549-8e73-6a6d1f60a71e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"
Jul 27 22:20:15.121878 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.121704    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/005e2639-7ed6-4794-aa8c-73cdaee9e580-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4\" (UID: \"005e2639-7ed6-4794-aa8c-73cdaee9e580\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"
Jul 27 22:20:15.121878 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.121812    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-588bd\" (UniqueName: \"kubernetes.io/projected/5da4ecd8-addc-4549-8e73-6a6d1f60a71e-kube-api-access-588bd\") pod \"payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx\" (UID: \"5da4ecd8-addc-4549-8e73-6a6d1f60a71e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"
Jul 27 22:20:15.121878 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.121847    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vs9\" (UniqueName: \"kubernetes.io/projected/005e2639-7ed6-4794-aa8c-73cdaee9e580-kube-api-access-p6vs9\") pod \"payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4\" (UID: \"005e2639-7ed6-4794-aa8c-73cdaee9e580\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"
Jul 27 22:20:15.183485 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.183450    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:20:15.222893 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.222858    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da4ecd8-addc-4549-8e73-6a6d1f60a71e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx\" (UID: \"5da4ecd8-addc-4549-8e73-6a6d1f60a71e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"
Jul 27 22:20:15.223082 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.222911    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/005e2639-7ed6-4794-aa8c-73cdaee9e580-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4\" (UID: \"005e2639-7ed6-4794-aa8c-73cdaee9e580\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"
Jul 27 22:20:15.223082 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.222957    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-588bd\" (UniqueName: \"kubernetes.io/projected/5da4ecd8-addc-4549-8e73-6a6d1f60a71e-kube-api-access-588bd\") pod \"payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx\" (UID: \"5da4ecd8-addc-4549-8e73-6a6d1f60a71e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"
Jul 27 22:20:15.223082 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.222983    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vs9\" (UniqueName: \"kubernetes.io/projected/005e2639-7ed6-4794-aa8c-73cdaee9e580-kube-api-access-p6vs9\") pod \"payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4\" (UID: \"005e2639-7ed6-4794-aa8c-73cdaee9e580\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"
Jul 27 22:20:15.223709 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.223689    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da4ecd8-addc-4549-8e73-6a6d1f60a71e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx\" (UID: \"5da4ecd8-addc-4549-8e73-6a6d1f60a71e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"
Jul 27 22:20:15.223761 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.223723    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/005e2639-7ed6-4794-aa8c-73cdaee9e580-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4\" (UID: \"005e2639-7ed6-4794-aa8c-73cdaee9e580\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"
Jul 27 22:20:15.235443 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.235414    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-588bd\" (UniqueName: \"kubernetes.io/projected/5da4ecd8-addc-4549-8e73-6a6d1f60a71e-kube-api-access-588bd\") pod \"payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx\" (UID: \"5da4ecd8-addc-4549-8e73-6a6d1f60a71e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"
Jul 27 22:20:15.237887 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.237858    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vs9\" (UniqueName: \"kubernetes.io/projected/005e2639-7ed6-4794-aa8c-73cdaee9e580-kube-api-access-p6vs9\") pod \"payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4\" (UID: \"005e2639-7ed6-4794-aa8c-73cdaee9e580\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"
Jul 27 22:20:15.303412 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.303368    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"
Jul 27 22:20:15.335560 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.335531    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"]
Jul 27 22:20:15.337040 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:20:15.337014    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ee2673_1eba_4089_a189_7b28c113f988.slice/crio-a72ea780bd9c3ced9028ef66891f33ceb861f82d635d0ccc4ecea50a0b10747f WatchSource:0}: Error finding container a72ea780bd9c3ced9028ef66891f33ceb861f82d635d0ccc4ecea50a0b10747f: Status 404 returned error can't find the container with id a72ea780bd9c3ced9028ef66891f33ceb861f82d635d0ccc4ecea50a0b10747f
Jul 27 22:20:15.342520 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.342493    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"
Jul 27 22:20:15.496866 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.496831    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"]
Jul 27 22:20:15.499519 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:20:15.499486    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da4ecd8_addc_4549_8e73_6a6d1f60a71e.slice/crio-d710e56e93dfa9e83296c7e97785db16e17300e0831f5f209cf8b5692181cc38 WatchSource:0}: Error finding container d710e56e93dfa9e83296c7e97785db16e17300e0831f5f209cf8b5692181cc38: Status 404 returned error can't find the container with id d710e56e93dfa9e83296c7e97785db16e17300e0831f5f209cf8b5692181cc38
Jul 27 22:20:15.664443 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.664393    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"]
Jul 27 22:20:15.666691 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:20:15.666661    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005e2639_7ed6_4794_aa8c_73cdaee9e580.slice/crio-85ad45987af95a530c2f8cd4009a5fdef621f6c3f9a8f5f0e6fa3dfbd1bd4352 WatchSource:0}: Error finding container 85ad45987af95a530c2f8cd4009a5fdef621f6c3f9a8f5f0e6fa3dfbd1bd4352: Status 404 returned error can't find the container with id 85ad45987af95a530c2f8cd4009a5fdef621f6c3f9a8f5f0e6fa3dfbd1bd4352
Jul 27 22:20:15.747811 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.747737    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:20:15.752788 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:15.752766    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:20:16.082513 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:16.082426    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx" event={"ID":"5da4ecd8-addc-4549-8e73-6a6d1f60a71e","Type":"ContainerStarted","Data":"7dcf36023a88be611e0a9421ea38f8dbb70ab6a458fd4795451388f81769977b"}
Jul 27 22:20:16.082513 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:16.082496    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx" event={"ID":"5da4ecd8-addc-4549-8e73-6a6d1f60a71e","Type":"ContainerStarted","Data":"d710e56e93dfa9e83296c7e97785db16e17300e0831f5f209cf8b5692181cc38"}
Jul 27 22:20:16.082730 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:16.082650    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"
Jul 27 22:20:16.084576 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:16.084544    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4" event={"ID":"005e2639-7ed6-4794-aa8c-73cdaee9e580","Type":"ContainerStarted","Data":"a8ea9f8d8d0a8d6b1e451fee31ae3518b32a53beedd6c548983c40facc53eb31"}
Jul 27 22:20:16.084731 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:16.084596    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4" event={"ID":"005e2639-7ed6-4794-aa8c-73cdaee9e580","Type":"ContainerStarted","Data":"85ad45987af95a530c2f8cd4009a5fdef621f6c3f9a8f5f0e6fa3dfbd1bd4352"}
Jul 27 22:20:16.084731 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:16.084624    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"
Jul 27 22:20:16.086492 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:16.086460    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7" event={"ID":"53ee2673-1eba-4089-a189-7b28c113f988","Type":"ContainerStarted","Data":"310f104ea3bfcb375daf274f1305eaa2a9c5bffab04f218527dfedbf282f2af5"}
Jul 27 22:20:16.086614 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:16.086508    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7" event={"ID":"53ee2673-1eba-4089-a189-7b28c113f988","Type":"ContainerStarted","Data":"a72ea780bd9c3ced9028ef66891f33ceb861f82d635d0ccc4ecea50a0b10747f"}
Jul 27 22:20:16.086614 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:16.086542    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:20:16.104625 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:16.104574    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx" podStartSLOduration=1.104557969 podStartE2EDuration="1.104557969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:20:16.101890762 +0000 UTC m=+2100.913238132" watchObservedRunningTime="2026-07-27 22:20:16.104557969 +0000 UTC m=+2100.915905340"
Jul 27 22:20:16.121726 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:16.121669    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7" podStartSLOduration=1.529407994 podStartE2EDuration="2.121654234s" totalImagesPullingTime="592.24624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:20:16.120408052 +0000 UTC m=+2100.931755428" watchObservedRunningTime="2026-07-27 22:20:16.121654234 +0000 UTC m=+2100.933001607"
Jul 27 22:20:16.137527 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:16.137481    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4" podStartSLOduration=2.137464268 podStartE2EDuration="2.137464268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:20:16.136205284 +0000 UTC m=+2100.947552658" watchObservedRunningTime="2026-07-27 22:20:16.137464268 +0000 UTC m=+2100.948811683"
Jul 27 22:20:22.098386 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:22.098354    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:20:22.442526 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:22.442490    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"]
Jul 27 22:20:22.446028 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:22.446000    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:20:22.449010 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:22.448984    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-00544d-serving-cert\""
Jul 27 22:20:22.457899 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:22.457873    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"]
Jul 27 22:20:22.593493 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:22.593455    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d98af-6bc4-4159-ae84-b840e1a17249-openshift-service-ca\") pod \"maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc\" (UID: \"ac0d98af-6bc4-4159-ae84-b840e1a17249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:20:22.593677 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:22.593503    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4lw\" (UniqueName: \"kubernetes.io/projected/ac0d98af-6bc4-4159-ae84-b840e1a17249-kube-api-access-kn4lw\") pod \"maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc\" (UID: \"ac0d98af-6bc4-4159-ae84-b840e1a17249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:20:22.593677 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:22.593535    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0d98af-6bc4-4159-ae84-b840e1a17249-maas-api-tls\") pod \"maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc\" (UID: \"ac0d98af-6bc4-4159-ae84-b840e1a17249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:20:22.700853 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:22.695131    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0d98af-6bc4-4159-ae84-b840e1a17249-maas-api-tls\") pod \"maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc\" (UID: \"ac0d98af-6bc4-4159-ae84-b840e1a17249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:20:22.700853 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:22.695296    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d98af-6bc4-4159-ae84-b840e1a17249-openshift-service-ca\") pod \"maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc\" (UID: \"ac0d98af-6bc4-4159-ae84-b840e1a17249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:20:22.700853 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:22.695349    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4lw\" (UniqueName: \"kubernetes.io/projected/ac0d98af-6bc4-4159-ae84-b840e1a17249-kube-api-access-kn4lw\") pod \"maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc\" (UID: \"ac0d98af-6bc4-4159-ae84-b840e1a17249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:20:22.700853 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:20:22.695811    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-serving-cert: secret "maas-api-e2e-inf-b-00544d-serving-cert" not found
Jul 27 22:20:22.700853 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:20:22.695891    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac0d98af-6bc4-4159-ae84-b840e1a17249-maas-api-tls podName:ac0d98af-6bc4-4159-ae84-b840e1a17249 nodeName:}" failed. No retries permitted until 2026-07-27 22:20:23.195869413 +0000 UTC m=+2108.007216788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac0d98af-6bc4-4159-ae84-b840e1a17249-maas-api-tls") pod "maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc" (UID: "ac0d98af-6bc4-4159-ae84-b840e1a17249") : secret "maas-api-e2e-inf-b-00544d-serving-cert" not found
Jul 27 22:20:22.700853 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:22.696762    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d98af-6bc4-4159-ae84-b840e1a17249-openshift-service-ca\") pod \"maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc\" (UID: \"ac0d98af-6bc4-4159-ae84-b840e1a17249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:20:22.707210 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:22.707175    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4lw\" (UniqueName: \"kubernetes.io/projected/ac0d98af-6bc4-4159-ae84-b840e1a17249-kube-api-access-kn4lw\") pod \"maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc\" (UID: \"ac0d98af-6bc4-4159-ae84-b840e1a17249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:20:23.081110 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.081028    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"]
Jul 27 22:20:23.084789 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.084767    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"
Jul 27 22:20:23.088418 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.088390    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-00544d-dockercfg-rjtrc\""
Jul 27 22:20:23.088778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.088762    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-00544d\""
Jul 27 22:20:23.100956 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.100929    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"]
Jul 27 22:20:23.121747 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.121705    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"]
Jul 27 22:20:23.125706 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.125684    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"
Jul 27 22:20:23.149290 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.149250    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"]
Jul 27 22:20:23.200576 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.200532    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66faaea4-870c-4e1f-bb6b-2b1a6c8c52be-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx\" (UID: \"66faaea4-870c-4e1f-bb6b-2b1a6c8c52be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"
Jul 27 22:20:23.200778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.200686    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88df803-b3a0-47f1-b9d3-14b6c083db49-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9\" (UID: \"d88df803-b3a0-47f1-b9d3-14b6c083db49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"
Jul 27 22:20:23.200778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.200725    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhdbh\" (UniqueName: \"kubernetes.io/projected/66faaea4-870c-4e1f-bb6b-2b1a6c8c52be-kube-api-access-vhdbh\") pod \"payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx\" (UID: \"66faaea4-870c-4e1f-bb6b-2b1a6c8c52be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"
Jul 27 22:20:23.200910 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.200794    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0d98af-6bc4-4159-ae84-b840e1a17249-maas-api-tls\") pod \"maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc\" (UID: \"ac0d98af-6bc4-4159-ae84-b840e1a17249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:20:23.200910 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.200854    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjcc9\" (UniqueName: \"kubernetes.io/projected/d88df803-b3a0-47f1-b9d3-14b6c083db49-kube-api-access-gjcc9\") pod \"payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9\" (UID: \"d88df803-b3a0-47f1-b9d3-14b6c083db49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"
Jul 27 22:20:23.203384 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.203362    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0d98af-6bc4-4159-ae84-b840e1a17249-maas-api-tls\") pod \"maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc\" (UID: \"ac0d98af-6bc4-4159-ae84-b840e1a17249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:20:23.301714 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.301668    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88df803-b3a0-47f1-b9d3-14b6c083db49-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9\" (UID: \"d88df803-b3a0-47f1-b9d3-14b6c083db49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"
Jul 27 22:20:23.301714 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.301715    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhdbh\" (UniqueName: \"kubernetes.io/projected/66faaea4-870c-4e1f-bb6b-2b1a6c8c52be-kube-api-access-vhdbh\") pod \"payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx\" (UID: \"66faaea4-870c-4e1f-bb6b-2b1a6c8c52be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"
Jul 27 22:20:23.301974 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.301756    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjcc9\" (UniqueName: \"kubernetes.io/projected/d88df803-b3a0-47f1-b9d3-14b6c083db49-kube-api-access-gjcc9\") pod \"payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9\" (UID: \"d88df803-b3a0-47f1-b9d3-14b6c083db49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"
Jul 27 22:20:23.301974 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.301792    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66faaea4-870c-4e1f-bb6b-2b1a6c8c52be-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx\" (UID: \"66faaea4-870c-4e1f-bb6b-2b1a6c8c52be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"
Jul 27 22:20:23.302549 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.302527    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66faaea4-870c-4e1f-bb6b-2b1a6c8c52be-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx\" (UID: \"66faaea4-870c-4e1f-bb6b-2b1a6c8c52be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"
Jul 27 22:20:23.302668 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.302646    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88df803-b3a0-47f1-b9d3-14b6c083db49-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9\" (UID: \"d88df803-b3a0-47f1-b9d3-14b6c083db49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"
Jul 27 22:20:23.313812 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.313787    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhdbh\" (UniqueName: \"kubernetes.io/projected/66faaea4-870c-4e1f-bb6b-2b1a6c8c52be-kube-api-access-vhdbh\") pod \"payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx\" (UID: \"66faaea4-870c-4e1f-bb6b-2b1a6c8c52be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"
Jul 27 22:20:23.315175 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.315153    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjcc9\" (UniqueName: \"kubernetes.io/projected/d88df803-b3a0-47f1-b9d3-14b6c083db49-kube-api-access-gjcc9\") pod \"payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9\" (UID: \"d88df803-b3a0-47f1-b9d3-14b6c083db49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"
Jul 27 22:20:23.359039 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.359007    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:20:23.397340 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.397271    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"
Jul 27 22:20:23.436624 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.436427    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"
Jul 27 22:20:23.534298 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.534193    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"]
Jul 27 22:20:23.536779 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:20:23.536692    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0d98af_6bc4_4159_ae84_b840e1a17249.slice/crio-fe90eaace0df8f7b15076f448199b06c3fbed9a266704ce35637ab36e031d12b WatchSource:0}: Error finding container fe90eaace0df8f7b15076f448199b06c3fbed9a266704ce35637ab36e031d12b: Status 404 returned error can't find the container with id fe90eaace0df8f7b15076f448199b06c3fbed9a266704ce35637ab36e031d12b
Jul 27 22:20:23.574105 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.574067    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"]
Jul 27 22:20:23.576811 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:20:23.576775    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88df803_b3a0_47f1_b9d3_14b6c083db49.slice/crio-8d1d297e3ebd5eb82d1c8b7bc16fb581b9306c20a8b0f609a85b6b3f1c31f8f4 WatchSource:0}: Error finding container 8d1d297e3ebd5eb82d1c8b7bc16fb581b9306c20a8b0f609a85b6b3f1c31f8f4: Status 404 returned error can't find the container with id 8d1d297e3ebd5eb82d1c8b7bc16fb581b9306c20a8b0f609a85b6b3f1c31f8f4
Jul 27 22:20:23.622997 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:23.622974    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"]
Jul 27 22:20:23.638436 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:20:23.638404    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66faaea4_870c_4e1f_bb6b_2b1a6c8c52be.slice/crio-c9710a5b29bede90db9632c6dcccefe1b1cc33cf4d96095c9598d0e021b31ce6 WatchSource:0}: Error finding container c9710a5b29bede90db9632c6dcccefe1b1cc33cf4d96095c9598d0e021b31ce6: Status 404 returned error can't find the container with id c9710a5b29bede90db9632c6dcccefe1b1cc33cf4d96095c9598d0e021b31ce6
Jul 27 22:20:24.134421 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:24.134384    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc" event={"ID":"ac0d98af-6bc4-4159-ae84-b840e1a17249","Type":"ContainerStarted","Data":"c64c8e450bfa393c9d4923506ef15d6dee8c98e0a6f79b9c579772a284cc49f2"}
Jul 27 22:20:24.134840 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:24.134443    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc" event={"ID":"ac0d98af-6bc4-4159-ae84-b840e1a17249","Type":"ContainerStarted","Data":"fe90eaace0df8f7b15076f448199b06c3fbed9a266704ce35637ab36e031d12b"}
Jul 27 22:20:24.134840 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:24.134546    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:20:24.135913 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:24.135889    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx" event={"ID":"66faaea4-870c-4e1f-bb6b-2b1a6c8c52be","Type":"ContainerStarted","Data":"300f53fcea2d5c926dc1a6152fbc9a060dcc6199e44a2776c0ddbc350902551e"}
Jul 27 22:20:24.136060 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:24.135937    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx" event={"ID":"66faaea4-870c-4e1f-bb6b-2b1a6c8c52be","Type":"ContainerStarted","Data":"c9710a5b29bede90db9632c6dcccefe1b1cc33cf4d96095c9598d0e021b31ce6"}
Jul 27 22:20:24.136060 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:24.135961    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"
Jul 27 22:20:24.137377 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:24.137355    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9" event={"ID":"d88df803-b3a0-47f1-b9d3-14b6c083db49","Type":"ContainerStarted","Data":"730cbb07ad9620a8de6117a46e91527a7c6437d0e66b8227717652675f22d1bb"}
Jul 27 22:20:24.137486 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:24.137391    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9" event={"ID":"d88df803-b3a0-47f1-b9d3-14b6c083db49","Type":"ContainerStarted","Data":"8d1d297e3ebd5eb82d1c8b7bc16fb581b9306c20a8b0f609a85b6b3f1c31f8f4"}
Jul 27 22:20:24.137547 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:24.137501    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"
Jul 27 22:20:24.155766 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:24.155719    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc" podStartSLOduration=1.786074875 podStartE2EDuration="2.155701304s" totalImagesPullingTime="369.626429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:20:24.153956754 +0000 UTC m=+2108.965304126" watchObservedRunningTime="2026-07-27 22:20:24.155701304 +0000 UTC m=+2108.967048676"
Jul 27 22:20:24.174289 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:24.174167    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx" podStartSLOduration=1.174144404 podStartE2EDuration="1.174144404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:20:24.171787288 +0000 UTC m=+2108.983134664" watchObservedRunningTime="2026-07-27 22:20:24.174144404 +0000 UTC m=+2108.985491781"
Jul 27 22:20:24.193277 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:24.193196    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9" podStartSLOduration=1.193178276 podStartE2EDuration="1.193178276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:20:24.1920638 +0000 UTC m=+2109.003411174" watchObservedRunningTime="2026-07-27 22:20:24.193178276 +0000 UTC m=+2109.004525682"
Jul 27 22:20:27.095296 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:27.095262    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"
Jul 27 22:20:27.095757 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:27.095439    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"
Jul 27 22:20:30.147204 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:30.147166    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:20:35.143619 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:35.143587    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"
Jul 27 22:20:35.144103 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:20:35.144024    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"
Jul 27 22:22:08.560581 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.560460    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"]
Jul 27 22:22:08.561384 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.560937    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7" podUID="53ee2673-1eba-4089-a189-7b28c113f988" containerName="maas-api" containerID="cri-o://310f104ea3bfcb375daf274f1305eaa2a9c5bffab04f218527dfedbf282f2af5" gracePeriod=30
Jul 27 22:22:08.633523 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.633483    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"]
Jul 27 22:22:08.633789 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.633737    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4" podUID="005e2639-7ed6-4794-aa8c-73cdaee9e580" containerName="payload-processing" containerID="cri-o://a8ea9f8d8d0a8d6b1e451fee31ae3518b32a53beedd6c548983c40facc53eb31" gracePeriod=30
Jul 27 22:22:08.642153 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.642067    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"]
Jul 27 22:22:08.642350 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.642321    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx" podUID="5da4ecd8-addc-4549-8e73-6a6d1f60a71e" containerName="payload-pre-processing" containerID="cri-o://7dcf36023a88be611e0a9421ea38f8dbb70ab6a458fd4795451388f81769977b" gracePeriod=30
Jul 27 22:22:08.904481 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.904449    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:22:08.938590 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.938439    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"
Jul 27 22:22:08.979233 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.979176    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"
Jul 27 22:22:08.979586 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.979504    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6vs9\" (UniqueName: \"kubernetes.io/projected/005e2639-7ed6-4794-aa8c-73cdaee9e580-kube-api-access-p6vs9\") pod \"005e2639-7ed6-4794-aa8c-73cdaee9e580\" (UID: \"005e2639-7ed6-4794-aa8c-73cdaee9e580\") "
Jul 27 22:22:08.979716 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.979594    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ee2673-1eba-4089-a189-7b28c113f988-openshift-service-ca\") pod \"53ee2673-1eba-4089-a189-7b28c113f988\" (UID: \"53ee2673-1eba-4089-a189-7b28c113f988\") "
Jul 27 22:22:08.979716 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.979624    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ee2673-1eba-4089-a189-7b28c113f988-maas-api-tls\") pod \"53ee2673-1eba-4089-a189-7b28c113f988\" (UID: \"53ee2673-1eba-4089-a189-7b28c113f988\") "
Jul 27 22:22:08.979716 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.979647    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/005e2639-7ed6-4794-aa8c-73cdaee9e580-plugins-config-volume\") pod \"005e2639-7ed6-4794-aa8c-73cdaee9e580\" (UID: \"005e2639-7ed6-4794-aa8c-73cdaee9e580\") "
Jul 27 22:22:08.979873 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.979720    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc4xp\" (UniqueName: \"kubernetes.io/projected/53ee2673-1eba-4089-a189-7b28c113f988-kube-api-access-jc4xp\") pod \"53ee2673-1eba-4089-a189-7b28c113f988\" (UID: \"53ee2673-1eba-4089-a189-7b28c113f988\") "
Jul 27 22:22:08.980387 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.980034    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ee2673-1eba-4089-a189-7b28c113f988-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "53ee2673-1eba-4089-a189-7b28c113f988" (UID: "53ee2673-1eba-4089-a189-7b28c113f988"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:22:08.980502 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.980485    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005e2639-7ed6-4794-aa8c-73cdaee9e580-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "005e2639-7ed6-4794-aa8c-73cdaee9e580" (UID: "005e2639-7ed6-4794-aa8c-73cdaee9e580"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:22:08.982906 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.982816    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005e2639-7ed6-4794-aa8c-73cdaee9e580-kube-api-access-p6vs9" (OuterVolumeSpecName: "kube-api-access-p6vs9") pod "005e2639-7ed6-4794-aa8c-73cdaee9e580" (UID: "005e2639-7ed6-4794-aa8c-73cdaee9e580"). InnerVolumeSpecName "kube-api-access-p6vs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:22:08.983059 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.982934    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ee2673-1eba-4089-a189-7b28c113f988-kube-api-access-jc4xp" (OuterVolumeSpecName: "kube-api-access-jc4xp") pod "53ee2673-1eba-4089-a189-7b28c113f988" (UID: "53ee2673-1eba-4089-a189-7b28c113f988"). InnerVolumeSpecName "kube-api-access-jc4xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:22:08.983059 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:08.982946    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ee2673-1eba-4089-a189-7b28c113f988-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "53ee2673-1eba-4089-a189-7b28c113f988" (UID: "53ee2673-1eba-4089-a189-7b28c113f988"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:22:09.080619 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.080520    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da4ecd8-addc-4549-8e73-6a6d1f60a71e-plugins-config-volume\") pod \"5da4ecd8-addc-4549-8e73-6a6d1f60a71e\" (UID: \"5da4ecd8-addc-4549-8e73-6a6d1f60a71e\") "
Jul 27 22:22:09.080619 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.080571    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-588bd\" (UniqueName: \"kubernetes.io/projected/5da4ecd8-addc-4549-8e73-6a6d1f60a71e-kube-api-access-588bd\") pod \"5da4ecd8-addc-4549-8e73-6a6d1f60a71e\" (UID: \"5da4ecd8-addc-4549-8e73-6a6d1f60a71e\") "
Jul 27 22:22:09.080943 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.080921    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ee2673-1eba-4089-a189-7b28c113f988-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:22:09.080984 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.080951    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ee2673-1eba-4089-a189-7b28c113f988-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:22:09.080984 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.080967    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/005e2639-7ed6-4794-aa8c-73cdaee9e580-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:22:09.081052 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.080974    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da4ecd8-addc-4549-8e73-6a6d1f60a71e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5da4ecd8-addc-4549-8e73-6a6d1f60a71e" (UID: "5da4ecd8-addc-4549-8e73-6a6d1f60a71e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:22:09.081052 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.080983    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc4xp\" (UniqueName: \"kubernetes.io/projected/53ee2673-1eba-4089-a189-7b28c113f988-kube-api-access-jc4xp\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:22:09.081052 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.081014    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6vs9\" (UniqueName: \"kubernetes.io/projected/005e2639-7ed6-4794-aa8c-73cdaee9e580-kube-api-access-p6vs9\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:22:09.083043 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.083015    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da4ecd8-addc-4549-8e73-6a6d1f60a71e-kube-api-access-588bd" (OuterVolumeSpecName: "kube-api-access-588bd") pod "5da4ecd8-addc-4549-8e73-6a6d1f60a71e" (UID: "5da4ecd8-addc-4549-8e73-6a6d1f60a71e"). InnerVolumeSpecName "kube-api-access-588bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:22:09.182392 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.182353    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da4ecd8-addc-4549-8e73-6a6d1f60a71e-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:22:09.182392 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.182391    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-588bd\" (UniqueName: \"kubernetes.io/projected/5da4ecd8-addc-4549-8e73-6a6d1f60a71e-kube-api-access-588bd\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:22:09.672723 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.672682    2549 generic.go:356] "Generic (PLEG): container finished" podID="5da4ecd8-addc-4549-8e73-6a6d1f60a71e" containerID="7dcf36023a88be611e0a9421ea38f8dbb70ab6a458fd4795451388f81769977b" exitCode=0
Jul 27 22:22:09.673203 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.672736    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"
Jul 27 22:22:09.673203 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.672776    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx" event={"ID":"5da4ecd8-addc-4549-8e73-6a6d1f60a71e","Type":"ContainerDied","Data":"7dcf36023a88be611e0a9421ea38f8dbb70ab6a458fd4795451388f81769977b"}
Jul 27 22:22:09.673203 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.672842    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx" event={"ID":"5da4ecd8-addc-4549-8e73-6a6d1f60a71e","Type":"ContainerDied","Data":"d710e56e93dfa9e83296c7e97785db16e17300e0831f5f209cf8b5692181cc38"}
Jul 27 22:22:09.673203 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.672882    2549 scope.go:117] "RemoveContainer" containerID="7dcf36023a88be611e0a9421ea38f8dbb70ab6a458fd4795451388f81769977b"
Jul 27 22:22:09.674193 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.674118    2549 generic.go:356] "Generic (PLEG): container finished" podID="005e2639-7ed6-4794-aa8c-73cdaee9e580" containerID="a8ea9f8d8d0a8d6b1e451fee31ae3518b32a53beedd6c548983c40facc53eb31" exitCode=0
Jul 27 22:22:09.674193 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.674166    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"
Jul 27 22:22:09.674322 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.674202    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4" event={"ID":"005e2639-7ed6-4794-aa8c-73cdaee9e580","Type":"ContainerDied","Data":"a8ea9f8d8d0a8d6b1e451fee31ae3518b32a53beedd6c548983c40facc53eb31"}
Jul 27 22:22:09.674322 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.674265    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4" event={"ID":"005e2639-7ed6-4794-aa8c-73cdaee9e580","Type":"ContainerDied","Data":"85ad45987af95a530c2f8cd4009a5fdef621f6c3f9a8f5f0e6fa3dfbd1bd4352"}
Jul 27 22:22:09.675766 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.675744    2549 generic.go:356] "Generic (PLEG): container finished" podID="53ee2673-1eba-4089-a189-7b28c113f988" containerID="310f104ea3bfcb375daf274f1305eaa2a9c5bffab04f218527dfedbf282f2af5" exitCode=0
Jul 27 22:22:09.675908 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.675785    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"
Jul 27 22:22:09.675908 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.675821    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7" event={"ID":"53ee2673-1eba-4089-a189-7b28c113f988","Type":"ContainerDied","Data":"310f104ea3bfcb375daf274f1305eaa2a9c5bffab04f218527dfedbf282f2af5"}
Jul 27 22:22:09.675908 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.675872    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7" event={"ID":"53ee2673-1eba-4089-a189-7b28c113f988","Type":"ContainerDied","Data":"a72ea780bd9c3ced9028ef66891f33ceb861f82d635d0ccc4ecea50a0b10747f"}
Jul 27 22:22:09.693075 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.693045    2549 scope.go:117] "RemoveContainer" containerID="7dcf36023a88be611e0a9421ea38f8dbb70ab6a458fd4795451388f81769977b"
Jul 27 22:22:09.693464 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:22:09.693428    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dcf36023a88be611e0a9421ea38f8dbb70ab6a458fd4795451388f81769977b\": container with ID starting with 7dcf36023a88be611e0a9421ea38f8dbb70ab6a458fd4795451388f81769977b not found: ID does not exist" containerID="7dcf36023a88be611e0a9421ea38f8dbb70ab6a458fd4795451388f81769977b"
Jul 27 22:22:09.693565 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.693468    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dcf36023a88be611e0a9421ea38f8dbb70ab6a458fd4795451388f81769977b"} err="failed to get container status \"7dcf36023a88be611e0a9421ea38f8dbb70ab6a458fd4795451388f81769977b\": rpc error: code = NotFound desc = could not find container \"7dcf36023a88be611e0a9421ea38f8dbb70ab6a458fd4795451388f81769977b\": container with ID starting with 7dcf36023a88be611e0a9421ea38f8dbb70ab6a458fd4795451388f81769977b not found: ID does not exist"
Jul 27 22:22:09.693565 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.693493    2549 scope.go:117] "RemoveContainer" containerID="a8ea9f8d8d0a8d6b1e451fee31ae3518b32a53beedd6c548983c40facc53eb31"
Jul 27 22:22:09.704563 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.704535    2549 scope.go:117] "RemoveContainer" containerID="a8ea9f8d8d0a8d6b1e451fee31ae3518b32a53beedd6c548983c40facc53eb31"
Jul 27 22:22:09.704956 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:22:09.704935    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ea9f8d8d0a8d6b1e451fee31ae3518b32a53beedd6c548983c40facc53eb31\": container with ID starting with a8ea9f8d8d0a8d6b1e451fee31ae3518b32a53beedd6c548983c40facc53eb31 not found: ID does not exist" containerID="a8ea9f8d8d0a8d6b1e451fee31ae3518b32a53beedd6c548983c40facc53eb31"
Jul 27 22:22:09.705028 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.704967    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ea9f8d8d0a8d6b1e451fee31ae3518b32a53beedd6c548983c40facc53eb31"} err="failed to get container status \"a8ea9f8d8d0a8d6b1e451fee31ae3518b32a53beedd6c548983c40facc53eb31\": rpc error: code = NotFound desc = could not find container \"a8ea9f8d8d0a8d6b1e451fee31ae3518b32a53beedd6c548983c40facc53eb31\": container with ID starting with a8ea9f8d8d0a8d6b1e451fee31ae3518b32a53beedd6c548983c40facc53eb31 not found: ID does not exist"
Jul 27 22:22:09.705028 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.704985    2549 scope.go:117] "RemoveContainer" containerID="310f104ea3bfcb375daf274f1305eaa2a9c5bffab04f218527dfedbf282f2af5"
Jul 27 22:22:09.713976 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.713661    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"]
Jul 27 22:22:09.713976 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.713878    2549 scope.go:117] "RemoveContainer" containerID="310f104ea3bfcb375daf274f1305eaa2a9c5bffab04f218527dfedbf282f2af5"
Jul 27 22:22:09.714307 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:22:09.714277    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310f104ea3bfcb375daf274f1305eaa2a9c5bffab04f218527dfedbf282f2af5\": container with ID starting with 310f104ea3bfcb375daf274f1305eaa2a9c5bffab04f218527dfedbf282f2af5 not found: ID does not exist" containerID="310f104ea3bfcb375daf274f1305eaa2a9c5bffab04f218527dfedbf282f2af5"
Jul 27 22:22:09.714416 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.714333    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310f104ea3bfcb375daf274f1305eaa2a9c5bffab04f218527dfedbf282f2af5"} err="failed to get container status \"310f104ea3bfcb375daf274f1305eaa2a9c5bffab04f218527dfedbf282f2af5\": rpc error: code = NotFound desc = could not find container \"310f104ea3bfcb375daf274f1305eaa2a9c5bffab04f218527dfedbf282f2af5\": container with ID starting with 310f104ea3bfcb375daf274f1305eaa2a9c5bffab04f218527dfedbf282f2af5 not found: ID does not exist"
Jul 27 22:22:09.716584 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.716565    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a6dea-65cc9cdd54-rtqp4"]
Jul 27 22:22:09.736658 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.736631    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"]
Jul 27 22:22:09.742398 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.742373    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a6dea-644bd7f7fd-r88b7"]
Jul 27 22:22:09.763276 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.763242    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"]
Jul 27 22:22:09.770803 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:09.770774    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a6dea-6957b49776-bfwlx"]
Jul 27 22:22:11.700736 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:11.700699    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="005e2639-7ed6-4794-aa8c-73cdaee9e580" path="/var/lib/kubelet/pods/005e2639-7ed6-4794-aa8c-73cdaee9e580/volumes"
Jul 27 22:22:11.701108 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:11.701091    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53ee2673-1eba-4089-a189-7b28c113f988" path="/var/lib/kubelet/pods/53ee2673-1eba-4089-a189-7b28c113f988/volumes"
Jul 27 22:22:11.701422 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:11.701409    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5da4ecd8-addc-4549-8e73-6a6d1f60a71e" path="/var/lib/kubelet/pods/5da4ecd8-addc-4549-8e73-6a6d1f60a71e/volumes"
Jul 27 22:22:30.695137 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:30.695104    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"]
Jul 27 22:22:30.695613 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:30.695326    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc" podUID="ac0d98af-6bc4-4159-ae84-b840e1a17249" containerName="maas-api" containerID="cri-o://c64c8e450bfa393c9d4923506ef15d6dee8c98e0a6f79b9c579772a284cc49f2" gracePeriod=30
Jul 27 22:22:30.734989 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:30.734955    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"]
Jul 27 22:22:30.735298 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:30.735217    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx" podUID="66faaea4-870c-4e1f-bb6b-2b1a6c8c52be" containerName="payload-processing" containerID="cri-o://300f53fcea2d5c926dc1a6152fbc9a060dcc6199e44a2776c0ddbc350902551e" gracePeriod=30
Jul 27 22:22:30.741425 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:30.741391    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"]
Jul 27 22:22:30.742009 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:30.741754    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9" podUID="d88df803-b3a0-47f1-b9d3-14b6c083db49" containerName="payload-pre-processing" containerID="cri-o://730cbb07ad9620a8de6117a46e91527a7c6437d0e66b8227717652675f22d1bb" gracePeriod=30
Jul 27 22:22:31.000406 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.000370    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"
Jul 27 22:22:31.050699 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.050670    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:22:31.053779 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.053755    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"
Jul 27 22:22:31.083590 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.083563    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66faaea4-870c-4e1f-bb6b-2b1a6c8c52be-plugins-config-volume\") pod \"66faaea4-870c-4e1f-bb6b-2b1a6c8c52be\" (UID: \"66faaea4-870c-4e1f-bb6b-2b1a6c8c52be\") "
Jul 27 22:22:31.083775 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.083657    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhdbh\" (UniqueName: \"kubernetes.io/projected/66faaea4-870c-4e1f-bb6b-2b1a6c8c52be-kube-api-access-vhdbh\") pod \"66faaea4-870c-4e1f-bb6b-2b1a6c8c52be\" (UID: \"66faaea4-870c-4e1f-bb6b-2b1a6c8c52be\") "
Jul 27 22:22:31.084011 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.083984    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66faaea4-870c-4e1f-bb6b-2b1a6c8c52be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66faaea4-870c-4e1f-bb6b-2b1a6c8c52be" (UID: "66faaea4-870c-4e1f-bb6b-2b1a6c8c52be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:22:31.086446 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.086421    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66faaea4-870c-4e1f-bb6b-2b1a6c8c52be-kube-api-access-vhdbh" (OuterVolumeSpecName: "kube-api-access-vhdbh") pod "66faaea4-870c-4e1f-bb6b-2b1a6c8c52be" (UID: "66faaea4-870c-4e1f-bb6b-2b1a6c8c52be"). InnerVolumeSpecName "kube-api-access-vhdbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:22:31.184772 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.184736    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d98af-6bc4-4159-ae84-b840e1a17249-openshift-service-ca\") pod \"ac0d98af-6bc4-4159-ae84-b840e1a17249\" (UID: \"ac0d98af-6bc4-4159-ae84-b840e1a17249\") "
Jul 27 22:22:31.184955 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.184824    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88df803-b3a0-47f1-b9d3-14b6c083db49-plugins-config-volume\") pod \"d88df803-b3a0-47f1-b9d3-14b6c083db49\" (UID: \"d88df803-b3a0-47f1-b9d3-14b6c083db49\") "
Jul 27 22:22:31.184955 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.184878    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0d98af-6bc4-4159-ae84-b840e1a17249-maas-api-tls\") pod \"ac0d98af-6bc4-4159-ae84-b840e1a17249\" (UID: \"ac0d98af-6bc4-4159-ae84-b840e1a17249\") "
Jul 27 22:22:31.184955 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.184935    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn4lw\" (UniqueName: \"kubernetes.io/projected/ac0d98af-6bc4-4159-ae84-b840e1a17249-kube-api-access-kn4lw\") pod \"ac0d98af-6bc4-4159-ae84-b840e1a17249\" (UID: \"ac0d98af-6bc4-4159-ae84-b840e1a17249\") "
Jul 27 22:22:31.185185 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.184974    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjcc9\" (UniqueName: \"kubernetes.io/projected/d88df803-b3a0-47f1-b9d3-14b6c083db49-kube-api-access-gjcc9\") pod \"d88df803-b3a0-47f1-b9d3-14b6c083db49\" (UID: \"d88df803-b3a0-47f1-b9d3-14b6c083db49\") "
Jul 27 22:22:31.185278 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.185167    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0d98af-6bc4-4159-ae84-b840e1a17249-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac0d98af-6bc4-4159-ae84-b840e1a17249" (UID: "ac0d98af-6bc4-4159-ae84-b840e1a17249"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:22:31.185278 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.185247    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88df803-b3a0-47f1-b9d3-14b6c083db49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d88df803-b3a0-47f1-b9d3-14b6c083db49" (UID: "d88df803-b3a0-47f1-b9d3-14b6c083db49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:22:31.185396 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.185312    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhdbh\" (UniqueName: \"kubernetes.io/projected/66faaea4-870c-4e1f-bb6b-2b1a6c8c52be-kube-api-access-vhdbh\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:22:31.185396 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.185334    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d98af-6bc4-4159-ae84-b840e1a17249-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:22:31.185396 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.185353    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66faaea4-870c-4e1f-bb6b-2b1a6c8c52be-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:22:31.187247 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.187200    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0d98af-6bc4-4159-ae84-b840e1a17249-kube-api-access-kn4lw" (OuterVolumeSpecName: "kube-api-access-kn4lw") pod "ac0d98af-6bc4-4159-ae84-b840e1a17249" (UID: "ac0d98af-6bc4-4159-ae84-b840e1a17249"). InnerVolumeSpecName "kube-api-access-kn4lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:22:31.187354 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.187282    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88df803-b3a0-47f1-b9d3-14b6c083db49-kube-api-access-gjcc9" (OuterVolumeSpecName: "kube-api-access-gjcc9") pod "d88df803-b3a0-47f1-b9d3-14b6c083db49" (UID: "d88df803-b3a0-47f1-b9d3-14b6c083db49"). InnerVolumeSpecName "kube-api-access-gjcc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:22:31.187354 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.187302    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0d98af-6bc4-4159-ae84-b840e1a17249-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac0d98af-6bc4-4159-ae84-b840e1a17249" (UID: "ac0d98af-6bc4-4159-ae84-b840e1a17249"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:22:31.285951 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.285859    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn4lw\" (UniqueName: \"kubernetes.io/projected/ac0d98af-6bc4-4159-ae84-b840e1a17249-kube-api-access-kn4lw\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:22:31.285951 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.285898    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjcc9\" (UniqueName: \"kubernetes.io/projected/d88df803-b3a0-47f1-b9d3-14b6c083db49-kube-api-access-gjcc9\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:22:31.285951 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.285913    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88df803-b3a0-47f1-b9d3-14b6c083db49-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:22:31.285951 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.285927    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0d98af-6bc4-4159-ae84-b840e1a17249-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:22:31.790635 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.790596    2549 generic.go:356] "Generic (PLEG): container finished" podID="ac0d98af-6bc4-4159-ae84-b840e1a17249" containerID="c64c8e450bfa393c9d4923506ef15d6dee8c98e0a6f79b9c579772a284cc49f2" exitCode=0
Jul 27 22:22:31.791114 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.790660    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"
Jul 27 22:22:31.791114 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.790689    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc" event={"ID":"ac0d98af-6bc4-4159-ae84-b840e1a17249","Type":"ContainerDied","Data":"c64c8e450bfa393c9d4923506ef15d6dee8c98e0a6f79b9c579772a284cc49f2"}
Jul 27 22:22:31.791114 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.790748    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc" event={"ID":"ac0d98af-6bc4-4159-ae84-b840e1a17249","Type":"ContainerDied","Data":"fe90eaace0df8f7b15076f448199b06c3fbed9a266704ce35637ab36e031d12b"}
Jul 27 22:22:31.791114 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.790765    2549 scope.go:117] "RemoveContainer" containerID="c64c8e450bfa393c9d4923506ef15d6dee8c98e0a6f79b9c579772a284cc49f2"
Jul 27 22:22:31.792175 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.791988    2549 generic.go:356] "Generic (PLEG): container finished" podID="66faaea4-870c-4e1f-bb6b-2b1a6c8c52be" containerID="300f53fcea2d5c926dc1a6152fbc9a060dcc6199e44a2776c0ddbc350902551e" exitCode=0
Jul 27 22:22:31.792175 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.792075    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"
Jul 27 22:22:31.792175 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.792085    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx" event={"ID":"66faaea4-870c-4e1f-bb6b-2b1a6c8c52be","Type":"ContainerDied","Data":"300f53fcea2d5c926dc1a6152fbc9a060dcc6199e44a2776c0ddbc350902551e"}
Jul 27 22:22:31.792175 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.792129    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx" event={"ID":"66faaea4-870c-4e1f-bb6b-2b1a6c8c52be","Type":"ContainerDied","Data":"c9710a5b29bede90db9632c6dcccefe1b1cc33cf4d96095c9598d0e021b31ce6"}
Jul 27 22:22:31.793779 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.793757    2549 generic.go:356] "Generic (PLEG): container finished" podID="d88df803-b3a0-47f1-b9d3-14b6c083db49" containerID="730cbb07ad9620a8de6117a46e91527a7c6437d0e66b8227717652675f22d1bb" exitCode=0
Jul 27 22:22:31.793894 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.793812    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"
Jul 27 22:22:31.793955 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.793813    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9" event={"ID":"d88df803-b3a0-47f1-b9d3-14b6c083db49","Type":"ContainerDied","Data":"730cbb07ad9620a8de6117a46e91527a7c6437d0e66b8227717652675f22d1bb"}
Jul 27 22:22:31.794013 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.793964    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9" event={"ID":"d88df803-b3a0-47f1-b9d3-14b6c083db49","Type":"ContainerDied","Data":"8d1d297e3ebd5eb82d1c8b7bc16fb581b9306c20a8b0f609a85b6b3f1c31f8f4"}
Jul 27 22:22:31.801058 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.801041    2549 scope.go:117] "RemoveContainer" containerID="c64c8e450bfa393c9d4923506ef15d6dee8c98e0a6f79b9c579772a284cc49f2"
Jul 27 22:22:31.801367 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:22:31.801344    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64c8e450bfa393c9d4923506ef15d6dee8c98e0a6f79b9c579772a284cc49f2\": container with ID starting with c64c8e450bfa393c9d4923506ef15d6dee8c98e0a6f79b9c579772a284cc49f2 not found: ID does not exist" containerID="c64c8e450bfa393c9d4923506ef15d6dee8c98e0a6f79b9c579772a284cc49f2"
Jul 27 22:22:31.801468 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.801373    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64c8e450bfa393c9d4923506ef15d6dee8c98e0a6f79b9c579772a284cc49f2"} err="failed to get container status \"c64c8e450bfa393c9d4923506ef15d6dee8c98e0a6f79b9c579772a284cc49f2\": rpc error: code = NotFound desc = could not find container \"c64c8e450bfa393c9d4923506ef15d6dee8c98e0a6f79b9c579772a284cc49f2\": container with ID starting with c64c8e450bfa393c9d4923506ef15d6dee8c98e0a6f79b9c579772a284cc49f2 not found: ID does not exist"
Jul 27 22:22:31.801468 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.801390    2549 scope.go:117] "RemoveContainer" containerID="300f53fcea2d5c926dc1a6152fbc9a060dcc6199e44a2776c0ddbc350902551e"
Jul 27 22:22:31.811934 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.811913    2549 scope.go:117] "RemoveContainer" containerID="300f53fcea2d5c926dc1a6152fbc9a060dcc6199e44a2776c0ddbc350902551e"
Jul 27 22:22:31.812258 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:22:31.812237    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300f53fcea2d5c926dc1a6152fbc9a060dcc6199e44a2776c0ddbc350902551e\": container with ID starting with 300f53fcea2d5c926dc1a6152fbc9a060dcc6199e44a2776c0ddbc350902551e not found: ID does not exist" containerID="300f53fcea2d5c926dc1a6152fbc9a060dcc6199e44a2776c0ddbc350902551e"
Jul 27 22:22:31.812331 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.812265    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300f53fcea2d5c926dc1a6152fbc9a060dcc6199e44a2776c0ddbc350902551e"} err="failed to get container status \"300f53fcea2d5c926dc1a6152fbc9a060dcc6199e44a2776c0ddbc350902551e\": rpc error: code = NotFound desc = could not find container \"300f53fcea2d5c926dc1a6152fbc9a060dcc6199e44a2776c0ddbc350902551e\": container with ID starting with 300f53fcea2d5c926dc1a6152fbc9a060dcc6199e44a2776c0ddbc350902551e not found: ID does not exist"
Jul 27 22:22:31.812331 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.812280    2549 scope.go:117] "RemoveContainer" containerID="730cbb07ad9620a8de6117a46e91527a7c6437d0e66b8227717652675f22d1bb"
Jul 27 22:22:31.820395 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.820374    2549 scope.go:117] "RemoveContainer" containerID="730cbb07ad9620a8de6117a46e91527a7c6437d0e66b8227717652675f22d1bb"
Jul 27 22:22:31.820751 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:22:31.820672    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730cbb07ad9620a8de6117a46e91527a7c6437d0e66b8227717652675f22d1bb\": container with ID starting with 730cbb07ad9620a8de6117a46e91527a7c6437d0e66b8227717652675f22d1bb not found: ID does not exist" containerID="730cbb07ad9620a8de6117a46e91527a7c6437d0e66b8227717652675f22d1bb"
Jul 27 22:22:31.820751 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.820706    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730cbb07ad9620a8de6117a46e91527a7c6437d0e66b8227717652675f22d1bb"} err="failed to get container status \"730cbb07ad9620a8de6117a46e91527a7c6437d0e66b8227717652675f22d1bb\": rpc error: code = NotFound desc = could not find container \"730cbb07ad9620a8de6117a46e91527a7c6437d0e66b8227717652675f22d1bb\": container with ID starting with 730cbb07ad9620a8de6117a46e91527a7c6437d0e66b8227717652675f22d1bb not found: ID does not exist"
Jul 27 22:22:31.822501 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.822481    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"]
Jul 27 22:22:31.828691 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.828668    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-00544d-89f55f45d-lskfx"]
Jul 27 22:22:31.846141 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.846117    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"]
Jul 27 22:22:31.853693 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.853671    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-00544d-5b955cf764-j58j9"]
Jul 27 22:22:31.868918 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.868889    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"]
Jul 27 22:22:31.876312 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:31.876281    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00544d-5cdf777468-7z9xc"]
Jul 27 22:22:33.701149 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:33.701113    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66faaea4-870c-4e1f-bb6b-2b1a6c8c52be" path="/var/lib/kubelet/pods/66faaea4-870c-4e1f-bb6b-2b1a6c8c52be/volumes"
Jul 27 22:22:33.701719 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:33.701696    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac0d98af-6bc4-4159-ae84-b840e1a17249" path="/var/lib/kubelet/pods/ac0d98af-6bc4-4159-ae84-b840e1a17249/volumes"
Jul 27 22:22:33.702139 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:33.702112    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d88df803-b3a0-47f1-b9d3-14b6c083db49" path="/var/lib/kubelet/pods/d88df803-b3a0-47f1-b9d3-14b6c083db49/volumes"
Jul 27 22:22:46.268523 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268444    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"]
Jul 27 22:22:46.268967 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268801    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d88df803-b3a0-47f1-b9d3-14b6c083db49" containerName="payload-pre-processing"
Jul 27 22:22:46.268967 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268812    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d88df803-b3a0-47f1-b9d3-14b6c083db49" containerName="payload-pre-processing"
Jul 27 22:22:46.268967 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268823    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ee2673-1eba-4089-a189-7b28c113f988" containerName="maas-api"
Jul 27 22:22:46.268967 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268829    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ee2673-1eba-4089-a189-7b28c113f988" containerName="maas-api"
Jul 27 22:22:46.268967 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268842    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac0d98af-6bc4-4159-ae84-b840e1a17249" containerName="maas-api"
Jul 27 22:22:46.268967 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268865    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac0d98af-6bc4-4159-ae84-b840e1a17249" containerName="maas-api"
Jul 27 22:22:46.268967 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268883    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66faaea4-870c-4e1f-bb6b-2b1a6c8c52be" containerName="payload-processing"
Jul 27 22:22:46.268967 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268889    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="66faaea4-870c-4e1f-bb6b-2b1a6c8c52be" containerName="payload-processing"
Jul 27 22:22:46.268967 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268900    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da4ecd8-addc-4549-8e73-6a6d1f60a71e" containerName="payload-pre-processing"
Jul 27 22:22:46.268967 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268906    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da4ecd8-addc-4549-8e73-6a6d1f60a71e" containerName="payload-pre-processing"
Jul 27 22:22:46.268967 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268912    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="005e2639-7ed6-4794-aa8c-73cdaee9e580" containerName="payload-processing"
Jul 27 22:22:46.268967 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268917    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="005e2639-7ed6-4794-aa8c-73cdaee9e580" containerName="payload-processing"
Jul 27 22:22:46.269357 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268977    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac0d98af-6bc4-4159-ae84-b840e1a17249" containerName="maas-api"
Jul 27 22:22:46.269357 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268983    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da4ecd8-addc-4549-8e73-6a6d1f60a71e" containerName="payload-pre-processing"
Jul 27 22:22:46.269357 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268991    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="53ee2673-1eba-4089-a189-7b28c113f988" containerName="maas-api"
Jul 27 22:22:46.269357 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.268996    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="66faaea4-870c-4e1f-bb6b-2b1a6c8c52be" containerName="payload-processing"
Jul 27 22:22:46.269357 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.269003    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="005e2639-7ed6-4794-aa8c-73cdaee9e580" containerName="payload-processing"
Jul 27 22:22:46.269357 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.269009    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d88df803-b3a0-47f1-b9d3-14b6c083db49" containerName="payload-pre-processing"
Jul 27 22:22:46.273711 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.273690    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:22:46.276444 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.276424    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-721ee8-serving-cert\""
Jul 27 22:22:46.283734 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.283705    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"]
Jul 27 22:22:46.317915 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.317685    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51c5e16-54f4-417c-9f71-ec3c7892a08e-maas-api-tls\") pod \"maas-api-e2e-api-a-721ee8-5685647db9-5fjzb\" (UID: \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:22:46.317915 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.317729    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4mzv\" (UniqueName: \"kubernetes.io/projected/f51c5e16-54f4-417c-9f71-ec3c7892a08e-kube-api-access-d4mzv\") pod \"maas-api-e2e-api-a-721ee8-5685647db9-5fjzb\" (UID: \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:22:46.317915 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.317795    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51c5e16-54f4-417c-9f71-ec3c7892a08e-openshift-service-ca\") pod \"maas-api-e2e-api-a-721ee8-5685647db9-5fjzb\" (UID: \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:22:46.418605 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.418560    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51c5e16-54f4-417c-9f71-ec3c7892a08e-maas-api-tls\") pod \"maas-api-e2e-api-a-721ee8-5685647db9-5fjzb\" (UID: \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:22:46.418605 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.418606    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4mzv\" (UniqueName: \"kubernetes.io/projected/f51c5e16-54f4-417c-9f71-ec3c7892a08e-kube-api-access-d4mzv\") pod \"maas-api-e2e-api-a-721ee8-5685647db9-5fjzb\" (UID: \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:22:46.418845 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.418649    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51c5e16-54f4-417c-9f71-ec3c7892a08e-openshift-service-ca\") pod \"maas-api-e2e-api-a-721ee8-5685647db9-5fjzb\" (UID: \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:22:46.418845 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:22:46.418717    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-serving-cert: secret "maas-api-e2e-api-a-721ee8-serving-cert" not found
Jul 27 22:22:46.418845 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:22:46.418784    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f51c5e16-54f4-417c-9f71-ec3c7892a08e-maas-api-tls podName:f51c5e16-54f4-417c-9f71-ec3c7892a08e nodeName:}" failed. No retries permitted until 2026-07-27 22:22:46.91876486 +0000 UTC m=+2251.730112210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f51c5e16-54f4-417c-9f71-ec3c7892a08e-maas-api-tls") pod "maas-api-e2e-api-a-721ee8-5685647db9-5fjzb" (UID: "f51c5e16-54f4-417c-9f71-ec3c7892a08e") : secret "maas-api-e2e-api-a-721ee8-serving-cert" not found
Jul 27 22:22:46.419387 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.419355    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51c5e16-54f4-417c-9f71-ec3c7892a08e-openshift-service-ca\") pod \"maas-api-e2e-api-a-721ee8-5685647db9-5fjzb\" (UID: \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:22:46.428874 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.428843    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4mzv\" (UniqueName: \"kubernetes.io/projected/f51c5e16-54f4-417c-9f71-ec3c7892a08e-kube-api-access-d4mzv\") pod \"maas-api-e2e-api-a-721ee8-5685647db9-5fjzb\" (UID: \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:22:46.922598 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.922562    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51c5e16-54f4-417c-9f71-ec3c7892a08e-maas-api-tls\") pod \"maas-api-e2e-api-a-721ee8-5685647db9-5fjzb\" (UID: \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:22:46.925184 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:46.925155    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51c5e16-54f4-417c-9f71-ec3c7892a08e-maas-api-tls\") pod \"maas-api-e2e-api-a-721ee8-5685647db9-5fjzb\" (UID: \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:22:47.180942 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.180857    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"]
Jul 27 22:22:47.184528 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.184502    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"
Jul 27 22:22:47.185632 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.185606    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:22:47.187944 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.187919    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-721ee8\""
Jul 27 22:22:47.188047 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.187942    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-721ee8-dockercfg-6jbbf\""
Jul 27 22:22:47.197665 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.197642    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"]
Jul 27 22:22:47.217166 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.217138    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"]
Jul 27 22:22:47.222081 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.222055    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"
Jul 27 22:22:47.224881 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.224856    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc7qx\" (UniqueName: \"kubernetes.io/projected/a0ec82ab-36c0-4307-a184-2d0273649cd1-kube-api-access-lc7qx\") pod \"payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk\" (UID: \"a0ec82ab-36c0-4307-a184-2d0273649cd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"
Jul 27 22:22:47.225026 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.224909    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ec82ab-36c0-4307-a184-2d0273649cd1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk\" (UID: \"a0ec82ab-36c0-4307-a184-2d0273649cd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"
Jul 27 22:22:47.225026 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.225003    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c26fe0d-9fe9-4dfd-a8dc-a67031431998-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4\" (UID: \"1c26fe0d-9fe9-4dfd-a8dc-a67031431998\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"
Jul 27 22:22:47.225125 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.225045    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5x7t\" (UniqueName: \"kubernetes.io/projected/1c26fe0d-9fe9-4dfd-a8dc-a67031431998-kube-api-access-v5x7t\") pod \"payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4\" (UID: \"1c26fe0d-9fe9-4dfd-a8dc-a67031431998\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"
Jul 27 22:22:47.238501 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.238479    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"]
Jul 27 22:22:47.325827 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.325789    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c26fe0d-9fe9-4dfd-a8dc-a67031431998-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4\" (UID: \"1c26fe0d-9fe9-4dfd-a8dc-a67031431998\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"
Jul 27 22:22:47.326304 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.325835    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5x7t\" (UniqueName: \"kubernetes.io/projected/1c26fe0d-9fe9-4dfd-a8dc-a67031431998-kube-api-access-v5x7t\") pod \"payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4\" (UID: \"1c26fe0d-9fe9-4dfd-a8dc-a67031431998\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"
Jul 27 22:22:47.326304 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.325889    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc7qx\" (UniqueName: \"kubernetes.io/projected/a0ec82ab-36c0-4307-a184-2d0273649cd1-kube-api-access-lc7qx\") pod \"payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk\" (UID: \"a0ec82ab-36c0-4307-a184-2d0273649cd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"
Jul 27 22:22:47.326304 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.325942    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ec82ab-36c0-4307-a184-2d0273649cd1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk\" (UID: \"a0ec82ab-36c0-4307-a184-2d0273649cd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"
Jul 27 22:22:47.326569 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.326549    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c26fe0d-9fe9-4dfd-a8dc-a67031431998-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4\" (UID: \"1c26fe0d-9fe9-4dfd-a8dc-a67031431998\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"
Jul 27 22:22:47.326855 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.326829    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ec82ab-36c0-4307-a184-2d0273649cd1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk\" (UID: \"a0ec82ab-36c0-4307-a184-2d0273649cd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"
Jul 27 22:22:47.337291 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.337266    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5x7t\" (UniqueName: \"kubernetes.io/projected/1c26fe0d-9fe9-4dfd-a8dc-a67031431998-kube-api-access-v5x7t\") pod \"payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4\" (UID: \"1c26fe0d-9fe9-4dfd-a8dc-a67031431998\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"
Jul 27 22:22:47.339336 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.339302    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc7qx\" (UniqueName: \"kubernetes.io/projected/a0ec82ab-36c0-4307-a184-2d0273649cd1-kube-api-access-lc7qx\") pod \"payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk\" (UID: \"a0ec82ab-36c0-4307-a184-2d0273649cd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"
Jul 27 22:22:47.357253 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.356473    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"]
Jul 27 22:22:47.362077 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:22:47.362049    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51c5e16_54f4_417c_9f71_ec3c7892a08e.slice/crio-605d6a54a6ae5b7c36411ed793a587c82690d13599e2c3c4b648f751a048874e WatchSource:0}: Error finding container 605d6a54a6ae5b7c36411ed793a587c82690d13599e2c3c4b648f751a048874e: Status 404 returned error can't find the container with id 605d6a54a6ae5b7c36411ed793a587c82690d13599e2c3c4b648f751a048874e
Jul 27 22:22:47.363451 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.363427    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 22:22:47.497554 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.497464    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"
Jul 27 22:22:47.534189 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.534151    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"
Jul 27 22:22:47.655619 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.655472    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"]
Jul 27 22:22:47.657717 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:22:47.657689    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ec82ab_36c0_4307_a184_2d0273649cd1.slice/crio-dc4f36702340012a78a2a4cb4d19e0fccebb1a0099293ec793b6a7ab093fceb6 WatchSource:0}: Error finding container dc4f36702340012a78a2a4cb4d19e0fccebb1a0099293ec793b6a7ab093fceb6: Status 404 returned error can't find the container with id dc4f36702340012a78a2a4cb4d19e0fccebb1a0099293ec793b6a7ab093fceb6
Jul 27 22:22:47.701400 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:22:47.701373    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c26fe0d_9fe9_4dfd_a8dc_a67031431998.slice/crio-3bd6077cc79bd883c3c2dfb2a739e5bb181d239380f137707fd04a59247ad6c8 WatchSource:0}: Error finding container 3bd6077cc79bd883c3c2dfb2a739e5bb181d239380f137707fd04a59247ad6c8: Status 404 returned error can't find the container with id 3bd6077cc79bd883c3c2dfb2a739e5bb181d239380f137707fd04a59247ad6c8
Jul 27 22:22:47.702157 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.702064    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"]
Jul 27 22:22:47.880423 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.880382    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb" event={"ID":"f51c5e16-54f4-417c-9f71-ec3c7892a08e","Type":"ContainerStarted","Data":"f9c2d0a0d4dd4a947eff820a2315e04cecda6f52c3c837f835ded350b0c6fd19"}
Jul 27 22:22:47.880620 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.880560    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:22:47.880620 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.880608    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb" event={"ID":"f51c5e16-54f4-417c-9f71-ec3c7892a08e","Type":"ContainerStarted","Data":"605d6a54a6ae5b7c36411ed793a587c82690d13599e2c3c4b648f751a048874e"}
Jul 27 22:22:47.881948 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.881924    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4" event={"ID":"1c26fe0d-9fe9-4dfd-a8dc-a67031431998","Type":"ContainerStarted","Data":"dbdf8fb2d74294b6cf022f79bad60ae772fbc41c70abc599c093ca26cb010803"}
Jul 27 22:22:47.882060 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.881965    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"
Jul 27 22:22:47.882060 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.881984    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4" event={"ID":"1c26fe0d-9fe9-4dfd-a8dc-a67031431998","Type":"ContainerStarted","Data":"3bd6077cc79bd883c3c2dfb2a739e5bb181d239380f137707fd04a59247ad6c8"}
Jul 27 22:22:47.883751 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.883724    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk" event={"ID":"a0ec82ab-36c0-4307-a184-2d0273649cd1","Type":"ContainerStarted","Data":"cb9369822fd21295da714af799e4cb27ba635b6490538474ffe846a36684a76b"}
Jul 27 22:22:47.883856 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.883770    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk" event={"ID":"a0ec82ab-36c0-4307-a184-2d0273649cd1","Type":"ContainerStarted","Data":"dc4f36702340012a78a2a4cb4d19e0fccebb1a0099293ec793b6a7ab093fceb6"}
Jul 27 22:22:47.884272 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.884250    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"
Jul 27 22:22:47.907702 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.907648    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb" podStartSLOduration=1.49067983 podStartE2EDuration="1.907629549s" totalImagesPullingTime="416.949719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:22:47.906599086 +0000 UTC m=+2252.717946458" watchObservedRunningTime="2026-07-27 22:22:47.907629549 +0000 UTC m=+2252.718976921"
Jul 27 22:22:47.930389 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.930338    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4" podStartSLOduration=0.93032191 podStartE2EDuration="930.32191ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:22:47.928915719 +0000 UTC m=+2252.740263092" watchObservedRunningTime="2026-07-27 22:22:47.93032191 +0000 UTC m=+2252.741669283"
Jul 27 22:22:47.950816 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:47.950766    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk" podStartSLOduration=0.950751853 podStartE2EDuration="950.751853ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:22:47.948919715 +0000 UTC m=+2252.760267089" watchObservedRunningTime="2026-07-27 22:22:47.950751853 +0000 UTC m=+2252.762099226"
Jul 27 22:22:53.896107 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:53.896076    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:22:55.455452 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:55.455418    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"]
Jul 27 22:22:55.458936 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:55.458911    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:22:55.462458 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:55.462436    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c75b14-serving-cert\""
Jul 27 22:22:55.475620 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:55.475588    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"]
Jul 27 22:22:55.496434 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:55.496240    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66385d4-84cc-42e3-917a-dc1d3ab787a6-maas-api-tls\") pod \"maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846\" (UID: \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:22:55.496434 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:55.496377    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msfxq\" (UniqueName: \"kubernetes.io/projected/e66385d4-84cc-42e3-917a-dc1d3ab787a6-kube-api-access-msfxq\") pod \"maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846\" (UID: \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:22:55.496434 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:55.496426    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66385d4-84cc-42e3-917a-dc1d3ab787a6-openshift-service-ca\") pod \"maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846\" (UID: \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:22:55.597477 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:55.597438    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msfxq\" (UniqueName: \"kubernetes.io/projected/e66385d4-84cc-42e3-917a-dc1d3ab787a6-kube-api-access-msfxq\") pod \"maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846\" (UID: \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:22:55.597658 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:55.597491    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66385d4-84cc-42e3-917a-dc1d3ab787a6-openshift-service-ca\") pod \"maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846\" (UID: \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:22:55.597658 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:55.597518    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66385d4-84cc-42e3-917a-dc1d3ab787a6-maas-api-tls\") pod \"maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846\" (UID: \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:22:55.597658 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:22:55.597636    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-serving-cert: secret "maas-api-e2e-api-b-c75b14-serving-cert" not found
Jul 27 22:22:55.597782 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:22:55.597693    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e66385d4-84cc-42e3-917a-dc1d3ab787a6-maas-api-tls podName:e66385d4-84cc-42e3-917a-dc1d3ab787a6 nodeName:}" failed. No retries permitted until 2026-07-27 22:22:56.097674159 +0000 UTC m=+2260.909021510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e66385d4-84cc-42e3-917a-dc1d3ab787a6-maas-api-tls") pod "maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846" (UID: "e66385d4-84cc-42e3-917a-dc1d3ab787a6") : secret "maas-api-e2e-api-b-c75b14-serving-cert" not found
Jul 27 22:22:55.598325 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:55.598301    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66385d4-84cc-42e3-917a-dc1d3ab787a6-openshift-service-ca\") pod \"maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846\" (UID: \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:22:55.619076 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:55.619051    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msfxq\" (UniqueName: \"kubernetes.io/projected/e66385d4-84cc-42e3-917a-dc1d3ab787a6-kube-api-access-msfxq\") pod \"maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846\" (UID: \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:22:56.102207 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.102163    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66385d4-84cc-42e3-917a-dc1d3ab787a6-maas-api-tls\") pod \"maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846\" (UID: \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:22:56.104879 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.104843    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66385d4-84cc-42e3-917a-dc1d3ab787a6-maas-api-tls\") pod \"maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846\" (UID: \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:22:56.247824 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.247789    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"]
Jul 27 22:22:56.251459 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.251418    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"
Jul 27 22:22:56.254564 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.254543    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c75b14\""
Jul 27 22:22:56.256170 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.256143    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c75b14-dockercfg-vps6k\""
Jul 27 22:22:56.264783 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.264756    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"]
Jul 27 22:22:56.303400 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.303358    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70be736d-89bd-408b-b926-b03dc0e0db60-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd\" (UID: \"70be736d-89bd-408b-b926-b03dc0e0db60\") " pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"
Jul 27 22:22:56.303400 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.303400    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5xv\" (UniqueName: \"kubernetes.io/projected/70be736d-89bd-408b-b926-b03dc0e0db60-kube-api-access-6z5xv\") pod \"payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd\" (UID: \"70be736d-89bd-408b-b926-b03dc0e0db60\") " pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"
Jul 27 22:22:56.326571 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.326531    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"]
Jul 27 22:22:56.330063 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.330044    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"
Jul 27 22:22:56.344788 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.344758    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"]
Jul 27 22:22:56.369849 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.369815    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:22:56.403957 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.403916    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70620f25-bbe8-45fd-aafd-612d328b00fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz\" (UID: \"70620f25-bbe8-45fd-aafd-612d328b00fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"
Jul 27 22:22:56.404095 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.403982    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70be736d-89bd-408b-b926-b03dc0e0db60-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd\" (UID: \"70be736d-89bd-408b-b926-b03dc0e0db60\") " pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"
Jul 27 22:22:56.404095 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.404005    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5xv\" (UniqueName: \"kubernetes.io/projected/70be736d-89bd-408b-b926-b03dc0e0db60-kube-api-access-6z5xv\") pod \"payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd\" (UID: \"70be736d-89bd-408b-b926-b03dc0e0db60\") " pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"
Jul 27 22:22:56.404095 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.404057    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vld7l\" (UniqueName: \"kubernetes.io/projected/70620f25-bbe8-45fd-aafd-612d328b00fa-kube-api-access-vld7l\") pod \"payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz\" (UID: \"70620f25-bbe8-45fd-aafd-612d328b00fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"
Jul 27 22:22:56.405004 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.404971    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70be736d-89bd-408b-b926-b03dc0e0db60-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd\" (UID: \"70be736d-89bd-408b-b926-b03dc0e0db60\") " pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"
Jul 27 22:22:56.417016 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.416947    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5xv\" (UniqueName: \"kubernetes.io/projected/70be736d-89bd-408b-b926-b03dc0e0db60-kube-api-access-6z5xv\") pod \"payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd\" (UID: \"70be736d-89bd-408b-b926-b03dc0e0db60\") " pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"
Jul 27 22:22:56.504836 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.504801    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vld7l\" (UniqueName: \"kubernetes.io/projected/70620f25-bbe8-45fd-aafd-612d328b00fa-kube-api-access-vld7l\") pod \"payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz\" (UID: \"70620f25-bbe8-45fd-aafd-612d328b00fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"
Jul 27 22:22:56.505234 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.504844    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70620f25-bbe8-45fd-aafd-612d328b00fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz\" (UID: \"70620f25-bbe8-45fd-aafd-612d328b00fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"
Jul 27 22:22:56.505570 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.505545    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70620f25-bbe8-45fd-aafd-612d328b00fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz\" (UID: \"70620f25-bbe8-45fd-aafd-612d328b00fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"
Jul 27 22:22:56.518740 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.518704    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vld7l\" (UniqueName: \"kubernetes.io/projected/70620f25-bbe8-45fd-aafd-612d328b00fa-kube-api-access-vld7l\") pod \"payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz\" (UID: \"70620f25-bbe8-45fd-aafd-612d328b00fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"
Jul 27 22:22:56.529831 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.529805    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"]
Jul 27 22:22:56.531523 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:22:56.531493    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66385d4_84cc_42e3_917a_dc1d3ab787a6.slice/crio-f1cf6e6088c0ffe13910fe2ba29b0d399467a16e1b31ba16b7b92fd28100931e WatchSource:0}: Error finding container f1cf6e6088c0ffe13910fe2ba29b0d399467a16e1b31ba16b7b92fd28100931e: Status 404 returned error can't find the container with id f1cf6e6088c0ffe13910fe2ba29b0d399467a16e1b31ba16b7b92fd28100931e
Jul 27 22:22:56.563160 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.563138    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"
Jul 27 22:22:56.640373 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.640293    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"
Jul 27 22:22:56.728379 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.728351    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"]
Jul 27 22:22:56.730011 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:22:56.729965    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70be736d_89bd_408b_b926_b03dc0e0db60.slice/crio-0c96aeaf2b5ad4fb4f19ecbd778b37f2367938d7f1c5d7a75d5529944ef16227 WatchSource:0}: Error finding container 0c96aeaf2b5ad4fb4f19ecbd778b37f2367938d7f1c5d7a75d5529944ef16227: Status 404 returned error can't find the container with id 0c96aeaf2b5ad4fb4f19ecbd778b37f2367938d7f1c5d7a75d5529944ef16227
Jul 27 22:22:56.811002 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:22:56.810964    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70620f25_bbe8_45fd_aafd_612d328b00fa.slice/crio-44944e481ac29899f1bfbc7ef94adc0d1ff91943c65ed8b1480d276bf0464877 WatchSource:0}: Error finding container 44944e481ac29899f1bfbc7ef94adc0d1ff91943c65ed8b1480d276bf0464877: Status 404 returned error can't find the container with id 44944e481ac29899f1bfbc7ef94adc0d1ff91943c65ed8b1480d276bf0464877
Jul 27 22:22:56.814609 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.814583    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"]
Jul 27 22:22:56.933990 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.933955    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd" event={"ID":"70be736d-89bd-408b-b926-b03dc0e0db60","Type":"ContainerStarted","Data":"4caaa9eb6a274825eb5a0a811f50f3867187ea198ab9a8e36886fe3fa03bb10e"}
Jul 27 22:22:56.934165 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.934017    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd" event={"ID":"70be736d-89bd-408b-b926-b03dc0e0db60","Type":"ContainerStarted","Data":"0c96aeaf2b5ad4fb4f19ecbd778b37f2367938d7f1c5d7a75d5529944ef16227"}
Jul 27 22:22:56.934165 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.934049    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"
Jul 27 22:22:56.935251 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.935207    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846" event={"ID":"e66385d4-84cc-42e3-917a-dc1d3ab787a6","Type":"ContainerStarted","Data":"f1cf6e6088c0ffe13910fe2ba29b0d399467a16e1b31ba16b7b92fd28100931e"}
Jul 27 22:22:56.936643 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.936622    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz" event={"ID":"70620f25-bbe8-45fd-aafd-612d328b00fa","Type":"ContainerStarted","Data":"470d62aeef2fe01c4047afdc894c009224a64c1d27a5b99b0ebb2d6557212a86"}
Jul 27 22:22:56.936753 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.936665    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz" event={"ID":"70620f25-bbe8-45fd-aafd-612d328b00fa","Type":"ContainerStarted","Data":"44944e481ac29899f1bfbc7ef94adc0d1ff91943c65ed8b1480d276bf0464877"}
Jul 27 22:22:56.936753 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.936692    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"
Jul 27 22:22:56.955303 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.955258    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd" podStartSLOduration=0.955242237 podStartE2EDuration="955.242237ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:22:56.95307101 +0000 UTC m=+2261.764418383" watchObservedRunningTime="2026-07-27 22:22:56.955242237 +0000 UTC m=+2261.766589623"
Jul 27 22:22:56.978122 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:56.978074    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz" podStartSLOduration=0.978059365 podStartE2EDuration="978.059365ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:22:56.976194635 +0000 UTC m=+2261.787542008" watchObservedRunningTime="2026-07-27 22:22:56.978059365 +0000 UTC m=+2261.789406737"
Jul 27 22:22:57.947087 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:57.947050    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846" event={"ID":"e66385d4-84cc-42e3-917a-dc1d3ab787a6","Type":"ContainerStarted","Data":"33c38fda0920e47dd6c19a386a40532b641c048c4f5fd6fc9601d752488f6f48"}
Jul 27 22:22:57.947566 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:57.947433    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:22:57.970413 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:57.970363    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846" podStartSLOduration=2.619105954 podStartE2EDuration="2.970347396s" totalImagesPullingTime="351.241442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:22:57.969148295 +0000 UTC m=+2262.780495680" watchObservedRunningTime="2026-07-27 22:22:57.970347396 +0000 UTC m=+2262.781694768"
Jul 27 22:22:58.892182 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:58.892124    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4" podUID="1c26fe0d-9fe9-4dfd-a8dc-a67031431998" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.70:9004: connect: connection refused"
Jul 27 22:22:59.899165 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:22:59.899132    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"
Jul 27 22:23:04.965293 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:04.965260    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:23:07.949347 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:07.949309    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"
Jul 27 22:23:07.949765 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:07.949499    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"
Jul 27 22:23:08.891025 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:08.890982    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"
Jul 27 22:23:23.798430 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:23.798395    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"]
Jul 27 22:23:23.798942 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:23.798605    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb" podUID="f51c5e16-54f4-417c-9f71-ec3c7892a08e" containerName="maas-api" containerID="cri-o://f9c2d0a0d4dd4a947eff820a2315e04cecda6f52c3c837f835ded350b0c6fd19" gracePeriod=30
Jul 27 22:23:23.851780 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:23.851742    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"]
Jul 27 22:23:23.852083 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:23.852024    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk" podUID="a0ec82ab-36c0-4307-a184-2d0273649cd1" containerName="payload-processing" containerID="cri-o://cb9369822fd21295da714af799e4cb27ba635b6490538474ffe846a36684a76b" gracePeriod=30
Jul 27 22:23:23.858586 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:23.858558    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"]
Jul 27 22:23:23.858822 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:23.858768    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4" podUID="1c26fe0d-9fe9-4dfd-a8dc-a67031431998" containerName="payload-pre-processing" containerID="cri-o://dbdf8fb2d74294b6cf022f79bad60ae772fbc41c70abc599c093ca26cb010803" gracePeriod=30
Jul 27 22:23:23.890510 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:23.890471    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb" podUID="f51c5e16-54f4-417c-9f71-ec3c7892a08e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.68:8443/health\": dial tcp 10.132.0.68:8443: connect: connection refused"
Jul 27 22:23:24.086796 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.086764    2549 generic.go:356] "Generic (PLEG): container finished" podID="a0ec82ab-36c0-4307-a184-2d0273649cd1" containerID="cb9369822fd21295da714af799e4cb27ba635b6490538474ffe846a36684a76b" exitCode=0
Jul 27 22:23:24.086984 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.086832    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk" event={"ID":"a0ec82ab-36c0-4307-a184-2d0273649cd1","Type":"ContainerDied","Data":"cb9369822fd21295da714af799e4cb27ba635b6490538474ffe846a36684a76b"}
Jul 27 22:23:24.088790 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.088764    2549 generic.go:356] "Generic (PLEG): container finished" podID="f51c5e16-54f4-417c-9f71-ec3c7892a08e" containerID="f9c2d0a0d4dd4a947eff820a2315e04cecda6f52c3c837f835ded350b0c6fd19" exitCode=0
Jul 27 22:23:24.088929 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.088843    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb" event={"ID":"f51c5e16-54f4-417c-9f71-ec3c7892a08e","Type":"ContainerDied","Data":"f9c2d0a0d4dd4a947eff820a2315e04cecda6f52c3c837f835ded350b0c6fd19"}
Jul 27 22:23:24.091745 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.091709    2549 generic.go:356] "Generic (PLEG): container finished" podID="1c26fe0d-9fe9-4dfd-a8dc-a67031431998" containerID="dbdf8fb2d74294b6cf022f79bad60ae772fbc41c70abc599c093ca26cb010803" exitCode=0
Jul 27 22:23:24.091841 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.091748    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4" event={"ID":"1c26fe0d-9fe9-4dfd-a8dc-a67031431998","Type":"ContainerDied","Data":"dbdf8fb2d74294b6cf022f79bad60ae772fbc41c70abc599c093ca26cb010803"}
Jul 27 22:23:24.135976 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.135948    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"
Jul 27 22:23:24.156092 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.155645    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:23:24.168948 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.168922    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"
Jul 27 22:23:24.281608 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.281567    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51c5e16-54f4-417c-9f71-ec3c7892a08e-maas-api-tls\") pod \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\" (UID: \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\") "
Jul 27 22:23:24.281608 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.281615    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51c5e16-54f4-417c-9f71-ec3c7892a08e-openshift-service-ca\") pod \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\" (UID: \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\") "
Jul 27 22:23:24.281864 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.281641    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc7qx\" (UniqueName: \"kubernetes.io/projected/a0ec82ab-36c0-4307-a184-2d0273649cd1-kube-api-access-lc7qx\") pod \"a0ec82ab-36c0-4307-a184-2d0273649cd1\" (UID: \"a0ec82ab-36c0-4307-a184-2d0273649cd1\") "
Jul 27 22:23:24.281864 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.281700    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ec82ab-36c0-4307-a184-2d0273649cd1-plugins-config-volume\") pod \"a0ec82ab-36c0-4307-a184-2d0273649cd1\" (UID: \"a0ec82ab-36c0-4307-a184-2d0273649cd1\") "
Jul 27 22:23:24.281864 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.281777    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4mzv\" (UniqueName: \"kubernetes.io/projected/f51c5e16-54f4-417c-9f71-ec3c7892a08e-kube-api-access-d4mzv\") pod \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\" (UID: \"f51c5e16-54f4-417c-9f71-ec3c7892a08e\") "
Jul 27 22:23:24.281864 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.281822    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c26fe0d-9fe9-4dfd-a8dc-a67031431998-plugins-config-volume\") pod \"1c26fe0d-9fe9-4dfd-a8dc-a67031431998\" (UID: \"1c26fe0d-9fe9-4dfd-a8dc-a67031431998\") "
Jul 27 22:23:24.281864 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.281863    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5x7t\" (UniqueName: \"kubernetes.io/projected/1c26fe0d-9fe9-4dfd-a8dc-a67031431998-kube-api-access-v5x7t\") pod \"1c26fe0d-9fe9-4dfd-a8dc-a67031431998\" (UID: \"1c26fe0d-9fe9-4dfd-a8dc-a67031431998\") "
Jul 27 22:23:24.282137 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.282030    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51c5e16-54f4-417c-9f71-ec3c7892a08e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f51c5e16-54f4-417c-9f71-ec3c7892a08e" (UID: "f51c5e16-54f4-417c-9f71-ec3c7892a08e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:23:24.282196 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.282174    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ec82ab-36c0-4307-a184-2d0273649cd1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0ec82ab-36c0-4307-a184-2d0273649cd1" (UID: "a0ec82ab-36c0-4307-a184-2d0273649cd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:23:24.282367 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.282323    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51c5e16-54f4-417c-9f71-ec3c7892a08e-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:23:24.282367 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.282350    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ec82ab-36c0-4307-a184-2d0273649cd1-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:23:24.282525 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.282498    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c26fe0d-9fe9-4dfd-a8dc-a67031431998-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c26fe0d-9fe9-4dfd-a8dc-a67031431998" (UID: "1c26fe0d-9fe9-4dfd-a8dc-a67031431998"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:23:24.284349 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.284315    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51c5e16-54f4-417c-9f71-ec3c7892a08e-kube-api-access-d4mzv" (OuterVolumeSpecName: "kube-api-access-d4mzv") pod "f51c5e16-54f4-417c-9f71-ec3c7892a08e" (UID: "f51c5e16-54f4-417c-9f71-ec3c7892a08e"). InnerVolumeSpecName "kube-api-access-d4mzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:23:24.284559 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.284537    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ec82ab-36c0-4307-a184-2d0273649cd1-kube-api-access-lc7qx" (OuterVolumeSpecName: "kube-api-access-lc7qx") pod "a0ec82ab-36c0-4307-a184-2d0273649cd1" (UID: "a0ec82ab-36c0-4307-a184-2d0273649cd1"). InnerVolumeSpecName "kube-api-access-lc7qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:23:24.284559 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.284546    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c26fe0d-9fe9-4dfd-a8dc-a67031431998-kube-api-access-v5x7t" (OuterVolumeSpecName: "kube-api-access-v5x7t") pod "1c26fe0d-9fe9-4dfd-a8dc-a67031431998" (UID: "1c26fe0d-9fe9-4dfd-a8dc-a67031431998"). InnerVolumeSpecName "kube-api-access-v5x7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:23:24.284763 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.284739    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f51c5e16-54f4-417c-9f71-ec3c7892a08e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f51c5e16-54f4-417c-9f71-ec3c7892a08e" (UID: "f51c5e16-54f4-417c-9f71-ec3c7892a08e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:23:24.382930 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.382894    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4mzv\" (UniqueName: \"kubernetes.io/projected/f51c5e16-54f4-417c-9f71-ec3c7892a08e-kube-api-access-d4mzv\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:23:24.382930 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.382928    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c26fe0d-9fe9-4dfd-a8dc-a67031431998-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:23:24.383141 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.382938    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5x7t\" (UniqueName: \"kubernetes.io/projected/1c26fe0d-9fe9-4dfd-a8dc-a67031431998-kube-api-access-v5x7t\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:23:24.383141 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.382947    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51c5e16-54f4-417c-9f71-ec3c7892a08e-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:23:24.383141 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:24.382958    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc7qx\" (UniqueName: \"kubernetes.io/projected/a0ec82ab-36c0-4307-a184-2d0273649cd1-kube-api-access-lc7qx\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:23:25.098715 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.098632    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"
Jul 27 22:23:25.099162 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.098629    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4" event={"ID":"1c26fe0d-9fe9-4dfd-a8dc-a67031431998","Type":"ContainerDied","Data":"3bd6077cc79bd883c3c2dfb2a739e5bb181d239380f137707fd04a59247ad6c8"}
Jul 27 22:23:25.099162 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.098772    2549 scope.go:117] "RemoveContainer" containerID="dbdf8fb2d74294b6cf022f79bad60ae772fbc41c70abc599c093ca26cb010803"
Jul 27 22:23:25.100100 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.099933    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"
Jul 27 22:23:25.100100 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.099961    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk" event={"ID":"a0ec82ab-36c0-4307-a184-2d0273649cd1","Type":"ContainerDied","Data":"dc4f36702340012a78a2a4cb4d19e0fccebb1a0099293ec793b6a7ab093fceb6"}
Jul 27 22:23:25.101535 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.101519    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"
Jul 27 22:23:25.101535 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.101522    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb" event={"ID":"f51c5e16-54f4-417c-9f71-ec3c7892a08e","Type":"ContainerDied","Data":"605d6a54a6ae5b7c36411ed793a587c82690d13599e2c3c4b648f751a048874e"}
Jul 27 22:23:25.108424 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.108408    2549 scope.go:117] "RemoveContainer" containerID="cb9369822fd21295da714af799e4cb27ba635b6490538474ffe846a36684a76b"
Jul 27 22:23:25.117458 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.117440    2549 scope.go:117] "RemoveContainer" containerID="f9c2d0a0d4dd4a947eff820a2315e04cecda6f52c3c837f835ded350b0c6fd19"
Jul 27 22:23:25.128235 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.128200    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"]
Jul 27 22:23:25.132202 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.132179    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-721ee8-6ff9db5648-wjjqk"]
Jul 27 22:23:25.144199 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.144176    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"]
Jul 27 22:23:25.149115 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.149095    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-721ee8-74cd6c6844-72wc4"]
Jul 27 22:23:25.162412 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.162387    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"]
Jul 27 22:23:25.168275 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.168252    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-721ee8-5685647db9-5fjzb"]
Jul 27 22:23:25.701779 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.701745    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c26fe0d-9fe9-4dfd-a8dc-a67031431998" path="/var/lib/kubelet/pods/1c26fe0d-9fe9-4dfd-a8dc-a67031431998/volumes"
Jul 27 22:23:25.702103 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.702090    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ec82ab-36c0-4307-a184-2d0273649cd1" path="/var/lib/kubelet/pods/a0ec82ab-36c0-4307-a184-2d0273649cd1/volumes"
Jul 27 22:23:25.702477 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:25.702463    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f51c5e16-54f4-417c-9f71-ec3c7892a08e" path="/var/lib/kubelet/pods/f51c5e16-54f4-417c-9f71-ec3c7892a08e/volumes"
Jul 27 22:23:51.679193 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:51.679155    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"]
Jul 27 22:23:51.679720 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:51.679450    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846" podUID="e66385d4-84cc-42e3-917a-dc1d3ab787a6" containerName="maas-api" containerID="cri-o://33c38fda0920e47dd6c19a386a40532b641c048c4f5fd6fc9601d752488f6f48" gracePeriod=30
Jul 27 22:23:51.737606 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:51.737570    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"]
Jul 27 22:23:51.737856 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:51.737805    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd" podUID="70be736d-89bd-408b-b926-b03dc0e0db60" containerName="payload-processing" containerID="cri-o://4caaa9eb6a274825eb5a0a811f50f3867187ea198ab9a8e36886fe3fa03bb10e" gracePeriod=30
Jul 27 22:23:51.744566 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:51.744524    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"]
Jul 27 22:23:51.744891 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:51.744759    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz" podUID="70620f25-bbe8-45fd-aafd-612d328b00fa" containerName="payload-pre-processing" containerID="cri-o://470d62aeef2fe01c4047afdc894c009224a64c1d27a5b99b0ebb2d6557212a86" gracePeriod=30
Jul 27 22:23:52.030202 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.030175    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:23:52.034008 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.033990    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"
Jul 27 22:23:52.052656 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.052632    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"
Jul 27 22:23:52.129353 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.129316    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66385d4-84cc-42e3-917a-dc1d3ab787a6-openshift-service-ca\") pod \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\" (UID: \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\") "
Jul 27 22:23:52.129522 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.129382    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z5xv\" (UniqueName: \"kubernetes.io/projected/70be736d-89bd-408b-b926-b03dc0e0db60-kube-api-access-6z5xv\") pod \"70be736d-89bd-408b-b926-b03dc0e0db60\" (UID: \"70be736d-89bd-408b-b926-b03dc0e0db60\") "
Jul 27 22:23:52.129522 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.129412    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vld7l\" (UniqueName: \"kubernetes.io/projected/70620f25-bbe8-45fd-aafd-612d328b00fa-kube-api-access-vld7l\") pod \"70620f25-bbe8-45fd-aafd-612d328b00fa\" (UID: \"70620f25-bbe8-45fd-aafd-612d328b00fa\") "
Jul 27 22:23:52.129522 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.129436    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66385d4-84cc-42e3-917a-dc1d3ab787a6-maas-api-tls\") pod \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\" (UID: \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\") "
Jul 27 22:23:52.129522 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.129459    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70be736d-89bd-408b-b926-b03dc0e0db60-plugins-config-volume\") pod \"70be736d-89bd-408b-b926-b03dc0e0db60\" (UID: \"70be736d-89bd-408b-b926-b03dc0e0db60\") "
Jul 27 22:23:52.129759 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.129614    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70620f25-bbe8-45fd-aafd-612d328b00fa-plugins-config-volume\") pod \"70620f25-bbe8-45fd-aafd-612d328b00fa\" (UID: \"70620f25-bbe8-45fd-aafd-612d328b00fa\") "
Jul 27 22:23:52.129759 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.129672    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msfxq\" (UniqueName: \"kubernetes.io/projected/e66385d4-84cc-42e3-917a-dc1d3ab787a6-kube-api-access-msfxq\") pod \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\" (UID: \"e66385d4-84cc-42e3-917a-dc1d3ab787a6\") "
Jul 27 22:23:52.129864 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.129768    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66385d4-84cc-42e3-917a-dc1d3ab787a6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e66385d4-84cc-42e3-917a-dc1d3ab787a6" (UID: "e66385d4-84cc-42e3-917a-dc1d3ab787a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:23:52.130019 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.129995    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66385d4-84cc-42e3-917a-dc1d3ab787a6-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:23:52.130151 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.130065    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70be736d-89bd-408b-b926-b03dc0e0db60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70be736d-89bd-408b-b926-b03dc0e0db60" (UID: "70be736d-89bd-408b-b926-b03dc0e0db60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:23:52.130151 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.130065    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70620f25-bbe8-45fd-aafd-612d328b00fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70620f25-bbe8-45fd-aafd-612d328b00fa" (UID: "70620f25-bbe8-45fd-aafd-612d328b00fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:23:52.131978 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.131952    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70620f25-bbe8-45fd-aafd-612d328b00fa-kube-api-access-vld7l" (OuterVolumeSpecName: "kube-api-access-vld7l") pod "70620f25-bbe8-45fd-aafd-612d328b00fa" (UID: "70620f25-bbe8-45fd-aafd-612d328b00fa"). InnerVolumeSpecName "kube-api-access-vld7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:23:52.131978 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.131964    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70be736d-89bd-408b-b926-b03dc0e0db60-kube-api-access-6z5xv" (OuterVolumeSpecName: "kube-api-access-6z5xv") pod "70be736d-89bd-408b-b926-b03dc0e0db60" (UID: "70be736d-89bd-408b-b926-b03dc0e0db60"). InnerVolumeSpecName "kube-api-access-6z5xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:23:52.132119 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.131963    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66385d4-84cc-42e3-917a-dc1d3ab787a6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e66385d4-84cc-42e3-917a-dc1d3ab787a6" (UID: "e66385d4-84cc-42e3-917a-dc1d3ab787a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:23:52.132403 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.132381    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66385d4-84cc-42e3-917a-dc1d3ab787a6-kube-api-access-msfxq" (OuterVolumeSpecName: "kube-api-access-msfxq") pod "e66385d4-84cc-42e3-917a-dc1d3ab787a6" (UID: "e66385d4-84cc-42e3-917a-dc1d3ab787a6"). InnerVolumeSpecName "kube-api-access-msfxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:23:52.231381 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.231289    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z5xv\" (UniqueName: \"kubernetes.io/projected/70be736d-89bd-408b-b926-b03dc0e0db60-kube-api-access-6z5xv\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:23:52.231381 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.231324    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vld7l\" (UniqueName: \"kubernetes.io/projected/70620f25-bbe8-45fd-aafd-612d328b00fa-kube-api-access-vld7l\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:23:52.231381 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.231335    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66385d4-84cc-42e3-917a-dc1d3ab787a6-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:23:52.231381 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.231343    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70be736d-89bd-408b-b926-b03dc0e0db60-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:23:52.231381 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.231352    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70620f25-bbe8-45fd-aafd-612d328b00fa-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:23:52.231381 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.231361    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msfxq\" (UniqueName: \"kubernetes.io/projected/e66385d4-84cc-42e3-917a-dc1d3ab787a6-kube-api-access-msfxq\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:23:52.241897 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.241855    2549 generic.go:356] "Generic (PLEG): container finished" podID="70be736d-89bd-408b-b926-b03dc0e0db60" containerID="4caaa9eb6a274825eb5a0a811f50f3867187ea198ab9a8e36886fe3fa03bb10e" exitCode=0
Jul 27 22:23:52.242056 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.241905    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"
Jul 27 22:23:52.242056 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.241916    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd" event={"ID":"70be736d-89bd-408b-b926-b03dc0e0db60","Type":"ContainerDied","Data":"4caaa9eb6a274825eb5a0a811f50f3867187ea198ab9a8e36886fe3fa03bb10e"}
Jul 27 22:23:52.242056 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.241974    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd" event={"ID":"70be736d-89bd-408b-b926-b03dc0e0db60","Type":"ContainerDied","Data":"0c96aeaf2b5ad4fb4f19ecbd778b37f2367938d7f1c5d7a75d5529944ef16227"}
Jul 27 22:23:52.242056 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.242005    2549 scope.go:117] "RemoveContainer" containerID="4caaa9eb6a274825eb5a0a811f50f3867187ea198ab9a8e36886fe3fa03bb10e"
Jul 27 22:23:52.243480 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.243454    2549 generic.go:356] "Generic (PLEG): container finished" podID="e66385d4-84cc-42e3-917a-dc1d3ab787a6" containerID="33c38fda0920e47dd6c19a386a40532b641c048c4f5fd6fc9601d752488f6f48" exitCode=0
Jul 27 22:23:52.243592 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.243502    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"
Jul 27 22:23:52.243592 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.243510    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846" event={"ID":"e66385d4-84cc-42e3-917a-dc1d3ab787a6","Type":"ContainerDied","Data":"33c38fda0920e47dd6c19a386a40532b641c048c4f5fd6fc9601d752488f6f48"}
Jul 27 22:23:52.243592 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.243583    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846" event={"ID":"e66385d4-84cc-42e3-917a-dc1d3ab787a6","Type":"ContainerDied","Data":"f1cf6e6088c0ffe13910fe2ba29b0d399467a16e1b31ba16b7b92fd28100931e"}
Jul 27 22:23:52.245684 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.245661    2549 generic.go:356] "Generic (PLEG): container finished" podID="70620f25-bbe8-45fd-aafd-612d328b00fa" containerID="470d62aeef2fe01c4047afdc894c009224a64c1d27a5b99b0ebb2d6557212a86" exitCode=0
Jul 27 22:23:52.245824 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.245690    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz" event={"ID":"70620f25-bbe8-45fd-aafd-612d328b00fa","Type":"ContainerDied","Data":"470d62aeef2fe01c4047afdc894c009224a64c1d27a5b99b0ebb2d6557212a86"}
Jul 27 22:23:52.245824 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.245711    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"
Jul 27 22:23:52.245824 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.245766    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz" event={"ID":"70620f25-bbe8-45fd-aafd-612d328b00fa","Type":"ContainerDied","Data":"44944e481ac29899f1bfbc7ef94adc0d1ff91943c65ed8b1480d276bf0464877"}
Jul 27 22:23:52.252826 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.252811    2549 scope.go:117] "RemoveContainer" containerID="4caaa9eb6a274825eb5a0a811f50f3867187ea198ab9a8e36886fe3fa03bb10e"
Jul 27 22:23:52.253126 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:23:52.253099    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4caaa9eb6a274825eb5a0a811f50f3867187ea198ab9a8e36886fe3fa03bb10e\": container with ID starting with 4caaa9eb6a274825eb5a0a811f50f3867187ea198ab9a8e36886fe3fa03bb10e not found: ID does not exist" containerID="4caaa9eb6a274825eb5a0a811f50f3867187ea198ab9a8e36886fe3fa03bb10e"
Jul 27 22:23:52.253245 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.253135    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4caaa9eb6a274825eb5a0a811f50f3867187ea198ab9a8e36886fe3fa03bb10e"} err="failed to get container status \"4caaa9eb6a274825eb5a0a811f50f3867187ea198ab9a8e36886fe3fa03bb10e\": rpc error: code = NotFound desc = could not find container \"4caaa9eb6a274825eb5a0a811f50f3867187ea198ab9a8e36886fe3fa03bb10e\": container with ID starting with 4caaa9eb6a274825eb5a0a811f50f3867187ea198ab9a8e36886fe3fa03bb10e not found: ID does not exist"
Jul 27 22:23:52.253245 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.253157    2549 scope.go:117] "RemoveContainer" containerID="33c38fda0920e47dd6c19a386a40532b641c048c4f5fd6fc9601d752488f6f48"
Jul 27 22:23:52.261396 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.261378    2549 scope.go:117] "RemoveContainer" containerID="33c38fda0920e47dd6c19a386a40532b641c048c4f5fd6fc9601d752488f6f48"
Jul 27 22:23:52.261651 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:23:52.261630    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c38fda0920e47dd6c19a386a40532b641c048c4f5fd6fc9601d752488f6f48\": container with ID starting with 33c38fda0920e47dd6c19a386a40532b641c048c4f5fd6fc9601d752488f6f48 not found: ID does not exist" containerID="33c38fda0920e47dd6c19a386a40532b641c048c4f5fd6fc9601d752488f6f48"
Jul 27 22:23:52.261711 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.261658    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c38fda0920e47dd6c19a386a40532b641c048c4f5fd6fc9601d752488f6f48"} err="failed to get container status \"33c38fda0920e47dd6c19a386a40532b641c048c4f5fd6fc9601d752488f6f48\": rpc error: code = NotFound desc = could not find container \"33c38fda0920e47dd6c19a386a40532b641c048c4f5fd6fc9601d752488f6f48\": container with ID starting with 33c38fda0920e47dd6c19a386a40532b641c048c4f5fd6fc9601d752488f6f48 not found: ID does not exist"
Jul 27 22:23:52.261711 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.261674    2549 scope.go:117] "RemoveContainer" containerID="470d62aeef2fe01c4047afdc894c009224a64c1d27a5b99b0ebb2d6557212a86"
Jul 27 22:23:52.269295 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.269272    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"]
Jul 27 22:23:52.269764 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.269751    2549 scope.go:117] "RemoveContainer" containerID="470d62aeef2fe01c4047afdc894c009224a64c1d27a5b99b0ebb2d6557212a86"
Jul 27 22:23:52.270047 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:23:52.270009    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470d62aeef2fe01c4047afdc894c009224a64c1d27a5b99b0ebb2d6557212a86\": container with ID starting with 470d62aeef2fe01c4047afdc894c009224a64c1d27a5b99b0ebb2d6557212a86 not found: ID does not exist" containerID="470d62aeef2fe01c4047afdc894c009224a64c1d27a5b99b0ebb2d6557212a86"
Jul 27 22:23:52.270142 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.270057    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470d62aeef2fe01c4047afdc894c009224a64c1d27a5b99b0ebb2d6557212a86"} err="failed to get container status \"470d62aeef2fe01c4047afdc894c009224a64c1d27a5b99b0ebb2d6557212a86\": rpc error: code = NotFound desc = could not find container \"470d62aeef2fe01c4047afdc894c009224a64c1d27a5b99b0ebb2d6557212a86\": container with ID starting with 470d62aeef2fe01c4047afdc894c009224a64c1d27a5b99b0ebb2d6557212a86 not found: ID does not exist"
Jul 27 22:23:52.276638 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.275990    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c75b14-679c9c59bb-ndxvd"]
Jul 27 22:23:52.289079 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.289051    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"]
Jul 27 22:23:52.293588 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.293565    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c75b14-7b98d59cb8-gj846"]
Jul 27 22:23:52.304313 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.304284    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"]
Jul 27 22:23:52.309717 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:52.309697    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c75b14-5f797c7ff6-2mzvz"]
Jul 27 22:23:53.701826 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:53.701783    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70620f25-bbe8-45fd-aafd-612d328b00fa" path="/var/lib/kubelet/pods/70620f25-bbe8-45fd-aafd-612d328b00fa/volumes"
Jul 27 22:23:53.702344 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:53.702325    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70be736d-89bd-408b-b926-b03dc0e0db60" path="/var/lib/kubelet/pods/70be736d-89bd-408b-b926-b03dc0e0db60/volumes"
Jul 27 22:23:53.702724 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:23:53.702705    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e66385d4-84cc-42e3-917a-dc1d3ab787a6" path="/var/lib/kubelet/pods/e66385d4-84cc-42e3-917a-dc1d3ab787a6/volumes"
Jul 27 22:24:06.568547 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.568458    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"]
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.568852    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70620f25-bbe8-45fd-aafd-612d328b00fa" containerName="payload-pre-processing"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.568865    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="70620f25-bbe8-45fd-aafd-612d328b00fa" containerName="payload-pre-processing"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.568878    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f51c5e16-54f4-417c-9f71-ec3c7892a08e" containerName="maas-api"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.568884    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f51c5e16-54f4-417c-9f71-ec3c7892a08e" containerName="maas-api"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.568890    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70be736d-89bd-408b-b926-b03dc0e0db60" containerName="payload-processing"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.568896    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="70be736d-89bd-408b-b926-b03dc0e0db60" containerName="payload-processing"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.568904    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66385d4-84cc-42e3-917a-dc1d3ab787a6" containerName="maas-api"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.568909    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66385d4-84cc-42e3-917a-dc1d3ab787a6" containerName="maas-api"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.568918    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c26fe0d-9fe9-4dfd-a8dc-a67031431998" containerName="payload-pre-processing"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.568923    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c26fe0d-9fe9-4dfd-a8dc-a67031431998" containerName="payload-pre-processing"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.568941    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ec82ab-36c0-4307-a184-2d0273649cd1" containerName="payload-processing"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.568946    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ec82ab-36c0-4307-a184-2d0273649cd1" containerName="payload-processing"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.569002    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ec82ab-36c0-4307-a184-2d0273649cd1" containerName="payload-processing"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.569010    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e66385d4-84cc-42e3-917a-dc1d3ab787a6" containerName="maas-api"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.569016    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c26fe0d-9fe9-4dfd-a8dc-a67031431998" containerName="payload-pre-processing"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.569021    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="f51c5e16-54f4-417c-9f71-ec3c7892a08e" containerName="maas-api"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.569027    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="70be736d-89bd-408b-b926-b03dc0e0db60" containerName="payload-processing"
Jul 27 22:24:06.569135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.569033    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="70620f25-bbe8-45fd-aafd-612d328b00fa" containerName="payload-pre-processing"
Jul 27 22:24:06.574126 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.574100    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:24:06.576884 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.576861    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-6138cc-serving-cert\""
Jul 27 22:24:06.583806 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.583781    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"]
Jul 27 22:24:06.756920 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.756881    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jskds\" (UniqueName: \"kubernetes.io/projected/61991ffa-5178-4f4d-bf85-0dbdf3b58882-kube-api-access-jskds\") pod \"maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn\" (UID: \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:24:06.756920 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.756922    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61991ffa-5178-4f4d-bf85-0dbdf3b58882-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn\" (UID: \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:24:06.757176 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.757013    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61991ffa-5178-4f4d-bf85-0dbdf3b58882-maas-api-tls\") pod \"maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn\" (UID: \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:24:06.857769 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.857730    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61991ffa-5178-4f4d-bf85-0dbdf3b58882-maas-api-tls\") pod \"maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn\" (UID: \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:24:06.857956 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.857821    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jskds\" (UniqueName: \"kubernetes.io/projected/61991ffa-5178-4f4d-bf85-0dbdf3b58882-kube-api-access-jskds\") pod \"maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn\" (UID: \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:24:06.857956 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.857854    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61991ffa-5178-4f4d-bf85-0dbdf3b58882-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn\" (UID: \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:24:06.858634 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.858569    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61991ffa-5178-4f4d-bf85-0dbdf3b58882-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn\" (UID: \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:24:06.860465 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.860443    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61991ffa-5178-4f4d-bf85-0dbdf3b58882-maas-api-tls\") pod \"maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn\" (UID: \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:24:06.868032 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.868005    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jskds\" (UniqueName: \"kubernetes.io/projected/61991ffa-5178-4f4d-bf85-0dbdf3b58882-kube-api-access-jskds\") pod \"maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn\" (UID: \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:24:06.885669 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:06.885635    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:24:07.030507 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.030441    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"]
Jul 27 22:24:07.033304 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:24:07.033269    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61991ffa_5178_4f4d_bf85_0dbdf3b58882.slice/crio-aee9764053f8b25e7e088d04eb031bc24acf828a6c20803d5f1bd66771907068 WatchSource:0}: Error finding container aee9764053f8b25e7e088d04eb031bc24acf828a6c20803d5f1bd66771907068: Status 404 returned error can't find the container with id aee9764053f8b25e7e088d04eb031bc24acf828a6c20803d5f1bd66771907068
Jul 27 22:24:07.320129 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.319870    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"]
Jul 27 22:24:07.325298 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.325262    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn" event={"ID":"61991ffa-5178-4f4d-bf85-0dbdf3b58882","Type":"ContainerStarted","Data":"aee9764053f8b25e7e088d04eb031bc24acf828a6c20803d5f1bd66771907068"}
Jul 27 22:24:07.325457 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.325362    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"
Jul 27 22:24:07.328526 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.328509    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-6138cc-dockercfg-2mgxt\""
Jul 27 22:24:07.328635 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.328508    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-6138cc\""
Jul 27 22:24:07.334391 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.334246    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"]
Jul 27 22:24:07.338118 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.338095    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"]
Jul 27 22:24:07.338268 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.338184    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"
Jul 27 22:24:07.349170 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.348847    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"]
Jul 27 22:24:07.464299 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.464264    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jg2l\" (UniqueName: \"kubernetes.io/projected/6bb37e2f-b130-4748-9414-cddb0ce4d922-kube-api-access-7jg2l\") pod \"payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22\" (UID: \"6bb37e2f-b130-4748-9414-cddb0ce4d922\") " pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"
Jul 27 22:24:07.464442 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.464345    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb37e2f-b130-4748-9414-cddb0ce4d922-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22\" (UID: \"6bb37e2f-b130-4748-9414-cddb0ce4d922\") " pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"
Jul 27 22:24:07.464442 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.464380    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th65p\" (UniqueName: \"kubernetes.io/projected/05987501-fdc8-4a57-92a6-62c587d609f3-kube-api-access-th65p\") pod \"payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm\" (UID: \"05987501-fdc8-4a57-92a6-62c587d609f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"
Jul 27 22:24:07.464533 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.464493    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05987501-fdc8-4a57-92a6-62c587d609f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm\" (UID: \"05987501-fdc8-4a57-92a6-62c587d609f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"
Jul 27 22:24:07.565164 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.565120    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05987501-fdc8-4a57-92a6-62c587d609f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm\" (UID: \"05987501-fdc8-4a57-92a6-62c587d609f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"
Jul 27 22:24:07.565164 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.565165    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jg2l\" (UniqueName: \"kubernetes.io/projected/6bb37e2f-b130-4748-9414-cddb0ce4d922-kube-api-access-7jg2l\") pod \"payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22\" (UID: \"6bb37e2f-b130-4748-9414-cddb0ce4d922\") " pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"
Jul 27 22:24:07.565481 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.565203    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb37e2f-b130-4748-9414-cddb0ce4d922-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22\" (UID: \"6bb37e2f-b130-4748-9414-cddb0ce4d922\") " pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"
Jul 27 22:24:07.565481 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.565259    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th65p\" (UniqueName: \"kubernetes.io/projected/05987501-fdc8-4a57-92a6-62c587d609f3-kube-api-access-th65p\") pod \"payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm\" (UID: \"05987501-fdc8-4a57-92a6-62c587d609f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"
Jul 27 22:24:07.565991 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.565967    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05987501-fdc8-4a57-92a6-62c587d609f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm\" (UID: \"05987501-fdc8-4a57-92a6-62c587d609f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"
Jul 27 22:24:07.566092 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.565995    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb37e2f-b130-4748-9414-cddb0ce4d922-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22\" (UID: \"6bb37e2f-b130-4748-9414-cddb0ce4d922\") " pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"
Jul 27 22:24:07.574898 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.574817    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jg2l\" (UniqueName: \"kubernetes.io/projected/6bb37e2f-b130-4748-9414-cddb0ce4d922-kube-api-access-7jg2l\") pod \"payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22\" (UID: \"6bb37e2f-b130-4748-9414-cddb0ce4d922\") " pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"
Jul 27 22:24:07.574898 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.574886    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th65p\" (UniqueName: \"kubernetes.io/projected/05987501-fdc8-4a57-92a6-62c587d609f3-kube-api-access-th65p\") pod \"payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm\" (UID: \"05987501-fdc8-4a57-92a6-62c587d609f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"
Jul 27 22:24:07.637841 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.637802    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"
Jul 27 22:24:07.649524 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.649496    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"
Jul 27 22:24:07.800462 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.800430    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"]
Jul 27 22:24:07.802829 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:24:07.802798    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05987501_fdc8_4a57_92a6_62c587d609f3.slice/crio-c1a4529b1ecfb0ba8e6ba4cab2f532f45dc39417471ba8b8708c71277a2dff2c WatchSource:0}: Error finding container c1a4529b1ecfb0ba8e6ba4cab2f532f45dc39417471ba8b8708c71277a2dff2c: Status 404 returned error can't find the container with id c1a4529b1ecfb0ba8e6ba4cab2f532f45dc39417471ba8b8708c71277a2dff2c
Jul 27 22:24:07.832282 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:07.832183    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"]
Jul 27 22:24:07.836422 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:24:07.836385    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb37e2f_b130_4748_9414_cddb0ce4d922.slice/crio-58bd55aaf9f28e6afee01d4cd8a6bb97b14115a7bfe2c669e9ec0dfff97cf810 WatchSource:0}: Error finding container 58bd55aaf9f28e6afee01d4cd8a6bb97b14115a7bfe2c669e9ec0dfff97cf810: Status 404 returned error can't find the container with id 58bd55aaf9f28e6afee01d4cd8a6bb97b14115a7bfe2c669e9ec0dfff97cf810
Jul 27 22:24:08.332141 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:08.332108    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm" event={"ID":"05987501-fdc8-4a57-92a6-62c587d609f3","Type":"ContainerStarted","Data":"367b18aef66e5ab284f52360549a8ac71a34813db201255660c8b2d843424113"}
Jul 27 22:24:08.332388 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:08.332169    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm" event={"ID":"05987501-fdc8-4a57-92a6-62c587d609f3","Type":"ContainerStarted","Data":"c1a4529b1ecfb0ba8e6ba4cab2f532f45dc39417471ba8b8708c71277a2dff2c"}
Jul 27 22:24:08.332388 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:08.332201    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"
Jul 27 22:24:08.333851 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:08.333823    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn" event={"ID":"61991ffa-5178-4f4d-bf85-0dbdf3b58882","Type":"ContainerStarted","Data":"c8e9c1cf62fb135207e2dc543e9404cb4e38148d8a84d043041f2fae7ab675a9"}
Jul 27 22:24:08.333980 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:08.333877    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:24:08.335359 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:08.335331    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22" event={"ID":"6bb37e2f-b130-4748-9414-cddb0ce4d922","Type":"ContainerStarted","Data":"1b91b10cd2d5cd00304710a195e07bc28648a1804a3ed833e6da46bc59f5c61a"}
Jul 27 22:24:08.335481 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:08.335382    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22" event={"ID":"6bb37e2f-b130-4748-9414-cddb0ce4d922","Type":"ContainerStarted","Data":"58bd55aaf9f28e6afee01d4cd8a6bb97b14115a7bfe2c669e9ec0dfff97cf810"}
Jul 27 22:24:08.335481 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:08.335411    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"
Jul 27 22:24:08.351455 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:08.351410    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm" podStartSLOduration=1.351395535 podStartE2EDuration="1.351395535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:24:08.349976003 +0000 UTC m=+2333.161323375" watchObservedRunningTime="2026-07-27 22:24:08.351395535 +0000 UTC m=+2333.162742907"
Jul 27 22:24:08.369261 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:08.369189    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn" podStartSLOduration=2.001326646 podStartE2EDuration="2.369174894s" totalImagesPullingTime="367.848248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:24:08.367710422 +0000 UTC m=+2333.179057796" watchObservedRunningTime="2026-07-27 22:24:08.369174894 +0000 UTC m=+2333.180522284"
Jul 27 22:24:08.388621 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:08.388567    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22" podStartSLOduration=1.388551914 podStartE2EDuration="1.388551914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:24:08.388009458 +0000 UTC m=+2333.199356834" watchObservedRunningTime="2026-07-27 22:24:08.388551914 +0000 UTC m=+2333.199899285"
Jul 27 22:24:14.346271 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:14.346217    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:24:14.597574 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:14.597498    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"]
Jul 27 22:24:14.601319 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:14.601297    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:24:14.604754 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:14.604730    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-0c79f1-serving-cert\""
Jul 27 22:24:14.611430 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:14.611401    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"]
Jul 27 22:24:14.651196 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:14.646636    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-maas-api-tls\") pod \"maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9\" (UID: \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:24:14.651196 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:14.646742    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9\" (UID: \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:24:14.651196 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:14.646804    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfbtv\" (UniqueName: \"kubernetes.io/projected/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-kube-api-access-vfbtv\") pod \"maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9\" (UID: \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:24:14.747549 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:14.747505    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-maas-api-tls\") pod \"maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9\" (UID: \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:24:14.747735 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:14.747565    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9\" (UID: \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:24:14.747735 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:14.747599    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfbtv\" (UniqueName: \"kubernetes.io/projected/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-kube-api-access-vfbtv\") pod \"maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9\" (UID: \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:24:14.747735 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:24:14.747680    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-serving-cert: secret "maas-api-e2e-shared-b-0c79f1-serving-cert" not found
Jul 27 22:24:14.747849 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:24:14.747776    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-maas-api-tls podName:8b8c63b8-69b4-4ca7-ac93-f5ac4807d251 nodeName:}" failed. No retries permitted until 2026-07-27 22:24:15.247753172 +0000 UTC m=+2340.059100524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-maas-api-tls") pod "maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9" (UID: "8b8c63b8-69b4-4ca7-ac93-f5ac4807d251") : secret "maas-api-e2e-shared-b-0c79f1-serving-cert" not found
Jul 27 22:24:14.748288 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:14.748270    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9\" (UID: \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:24:14.757209 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:14.757179    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfbtv\" (UniqueName: \"kubernetes.io/projected/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-kube-api-access-vfbtv\") pod \"maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9\" (UID: \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:24:15.251892 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.251854    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-maas-api-tls\") pod \"maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9\" (UID: \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:24:15.254472 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.254446    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-maas-api-tls\") pod \"maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9\" (UID: \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:24:15.361736 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.361696    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"]
Jul 27 22:24:15.365387 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.365362    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"
Jul 27 22:24:15.368304 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.368271    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-0c79f1\""
Jul 27 22:24:15.368304 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.368290    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-0c79f1-dockercfg-5p7m7\""
Jul 27 22:24:15.382584 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.382549    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"]
Jul 27 22:24:15.405688 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.405648    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"]
Jul 27 22:24:15.409043 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.409014    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"
Jul 27 22:24:15.420487 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.420456    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"]
Jul 27 22:24:15.453262 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.453210    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f5bj\" (UniqueName: \"kubernetes.io/projected/10c18dcd-919f-43ea-979b-c32d70922ad0-kube-api-access-2f5bj\") pod \"payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq\" (UID: \"10c18dcd-919f-43ea-979b-c32d70922ad0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"
Jul 27 22:24:15.453449 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.453344    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0488a54-696d-420c-b60d-c7a4f32de184-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw\" (UID: \"a0488a54-696d-420c-b60d-c7a4f32de184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"
Jul 27 22:24:15.453449 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.453373    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfjhk\" (UniqueName: \"kubernetes.io/projected/a0488a54-696d-420c-b60d-c7a4f32de184-kube-api-access-zfjhk\") pod \"payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw\" (UID: \"a0488a54-696d-420c-b60d-c7a4f32de184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"
Jul 27 22:24:15.453449 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.453404    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c18dcd-919f-43ea-979b-c32d70922ad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq\" (UID: \"10c18dcd-919f-43ea-979b-c32d70922ad0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"
Jul 27 22:24:15.513101 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.513014    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:24:15.554943 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.554903    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0488a54-696d-420c-b60d-c7a4f32de184-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw\" (UID: \"a0488a54-696d-420c-b60d-c7a4f32de184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"
Jul 27 22:24:15.555133 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.554955    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfjhk\" (UniqueName: \"kubernetes.io/projected/a0488a54-696d-420c-b60d-c7a4f32de184-kube-api-access-zfjhk\") pod \"payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw\" (UID: \"a0488a54-696d-420c-b60d-c7a4f32de184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"
Jul 27 22:24:15.555133 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.555012    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c18dcd-919f-43ea-979b-c32d70922ad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq\" (UID: \"10c18dcd-919f-43ea-979b-c32d70922ad0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"
Jul 27 22:24:15.555133 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.555114    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f5bj\" (UniqueName: \"kubernetes.io/projected/10c18dcd-919f-43ea-979b-c32d70922ad0-kube-api-access-2f5bj\") pod \"payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq\" (UID: \"10c18dcd-919f-43ea-979b-c32d70922ad0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"
Jul 27 22:24:15.557713 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.557690    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-0c79f1\""
Jul 27 22:24:15.564471 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.564446    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f5bj\" (UniqueName: \"kubernetes.io/projected/10c18dcd-919f-43ea-979b-c32d70922ad0-kube-api-access-2f5bj\") pod \"payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq\" (UID: \"10c18dcd-919f-43ea-979b-c32d70922ad0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"
Jul 27 22:24:15.564794 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.564771    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfjhk\" (UniqueName: \"kubernetes.io/projected/a0488a54-696d-420c-b60d-c7a4f32de184-kube-api-access-zfjhk\") pod \"payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw\" (UID: \"a0488a54-696d-420c-b60d-c7a4f32de184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"
Jul 27 22:24:15.565969 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.565938    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c18dcd-919f-43ea-979b-c32d70922ad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq\" (UID: \"10c18dcd-919f-43ea-979b-c32d70922ad0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"
Jul 27 22:24:15.566126 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.565991    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0488a54-696d-420c-b60d-c7a4f32de184-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw\" (UID: \"a0488a54-696d-420c-b60d-c7a4f32de184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"
Jul 27 22:24:15.674936 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.674909    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"]
Jul 27 22:24:15.677054 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:24:15.677019    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8c63b8_69b4_4ca7_ac93_f5ac4807d251.slice/crio-b8d082a1a72ef91eace31775a76164c7be41e7a093352c29aaabd6a0ef52b94b WatchSource:0}: Error finding container b8d082a1a72ef91eace31775a76164c7be41e7a093352c29aaabd6a0ef52b94b: Status 404 returned error can't find the container with id b8d082a1a72ef91eace31775a76164c7be41e7a093352c29aaabd6a0ef52b94b
Jul 27 22:24:15.679788 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.679768    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-0c79f1-dockercfg-5p7m7\""
Jul 27 22:24:15.687383 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.687361    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"
Jul 27 22:24:15.720735 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.720709    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"
Jul 27 22:24:15.854236 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.854188    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"]
Jul 27 22:24:15.856293 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:24:15.856214    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0488a54_696d_420c_b60d_c7a4f32de184.slice/crio-9ad3e2b63642bc51eada845a5c82dd0c1bfef484db7ad18c372a2b1efb46f975 WatchSource:0}: Error finding container 9ad3e2b63642bc51eada845a5c82dd0c1bfef484db7ad18c372a2b1efb46f975: Status 404 returned error can't find the container with id 9ad3e2b63642bc51eada845a5c82dd0c1bfef484db7ad18c372a2b1efb46f975
Jul 27 22:24:15.890021 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:15.889997    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"]
Jul 27 22:24:15.895500 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:24:15.895476    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c18dcd_919f_43ea_979b_c32d70922ad0.slice/crio-777d212f635ee4657750b4f209047cb3f77e45db596e1e8efaf6bddd2cf7bd11 WatchSource:0}: Error finding container 777d212f635ee4657750b4f209047cb3f77e45db596e1e8efaf6bddd2cf7bd11: Status 404 returned error can't find the container with id 777d212f635ee4657750b4f209047cb3f77e45db596e1e8efaf6bddd2cf7bd11
Jul 27 22:24:16.380732 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:16.380694    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq" event={"ID":"10c18dcd-919f-43ea-979b-c32d70922ad0","Type":"ContainerStarted","Data":"791aaa45ea8f65111090ad80c7b918a48378813baa327fda8cd09832006e5582"}
Jul 27 22:24:16.381210 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:16.380770    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"
Jul 27 22:24:16.381210 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:16.380799    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq" event={"ID":"10c18dcd-919f-43ea-979b-c32d70922ad0","Type":"ContainerStarted","Data":"777d212f635ee4657750b4f209047cb3f77e45db596e1e8efaf6bddd2cf7bd11"}
Jul 27 22:24:16.382496 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:16.382465    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9" event={"ID":"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251","Type":"ContainerStarted","Data":"f7e3f5767ca1cfdf516ac5c86194cd6ac43acbf31a6b1dd42fc6db6a1d034d5f"}
Jul 27 22:24:16.382632 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:16.382515    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9" event={"ID":"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251","Type":"ContainerStarted","Data":"b8d082a1a72ef91eace31775a76164c7be41e7a093352c29aaabd6a0ef52b94b"}
Jul 27 22:24:16.382994 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:16.382927    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:24:16.385349 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:16.385323    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw" event={"ID":"a0488a54-696d-420c-b60d-c7a4f32de184","Type":"ContainerStarted","Data":"32a49c91b9590573f53d43986b1e93e7cba2d161c91a7ea3c7715708c648dee8"}
Jul 27 22:24:16.385485 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:16.385370    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw" event={"ID":"a0488a54-696d-420c-b60d-c7a4f32de184","Type":"ContainerStarted","Data":"9ad3e2b63642bc51eada845a5c82dd0c1bfef484db7ad18c372a2b1efb46f975"}
Jul 27 22:24:16.385556 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:16.385480    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"
Jul 27 22:24:16.401922 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:16.401876    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq" podStartSLOduration=1.4018615589999999 podStartE2EDuration="1.401861559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:24:16.399185903 +0000 UTC m=+2341.210533311" watchObservedRunningTime="2026-07-27 22:24:16.401861559 +0000 UTC m=+2341.213208930"
Jul 27 22:24:16.435565 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:16.435510    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9" podStartSLOduration=2.053549424 podStartE2EDuration="2.435493376s" totalImagesPullingTime="381.943952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:24:16.43446686 +0000 UTC m=+2341.245814233" watchObservedRunningTime="2026-07-27 22:24:16.435493376 +0000 UTC m=+2341.246840747"
Jul 27 22:24:16.435738 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:16.435594    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw" podStartSLOduration=1.435589021 podStartE2EDuration="1.435589021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:24:16.417488495 +0000 UTC m=+2341.228835867" watchObservedRunningTime="2026-07-27 22:24:16.435589021 +0000 UTC m=+2341.246936392"
Jul 27 22:24:19.343033 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:19.343002    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"
Jul 27 22:24:19.343419 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:19.343080    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"
Jul 27 22:24:23.402135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:23.402102    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:24:27.393565 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:27.393532    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"
Jul 27 22:24:27.393939 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:24:27.393610    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"
Jul 27 22:25:15.779405 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:25:15.779365    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:25:15.787213 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:25:15.787187    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:30:15.809469 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:30:15.809437    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:30:15.818778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:30:15.818750    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:34:00.705545 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:00.705509    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"]
Jul 27 22:34:00.709444 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:00.709420    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:34:00.712879 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:00.712856    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-626801-serving-cert\""
Jul 27 22:34:00.721047 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:00.721019    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"]
Jul 27 22:34:00.879202 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:00.879164    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2889f9a4-a913-482c-b59d-f51d4a5a2955-maas-api-tls\") pod \"maas-api-e2e-ipp-a-626801-6f787498ff-hq648\" (UID: \"2889f9a4-a913-482c-b59d-f51d4a5a2955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:34:00.879423 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:00.879263    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-849cn\" (UniqueName: \"kubernetes.io/projected/2889f9a4-a913-482c-b59d-f51d4a5a2955-kube-api-access-849cn\") pod \"maas-api-e2e-ipp-a-626801-6f787498ff-hq648\" (UID: \"2889f9a4-a913-482c-b59d-f51d4a5a2955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:34:00.879423 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:00.879294    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2889f9a4-a913-482c-b59d-f51d4a5a2955-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-626801-6f787498ff-hq648\" (UID: \"2889f9a4-a913-482c-b59d-f51d4a5a2955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:34:00.980374 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:00.980268    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2889f9a4-a913-482c-b59d-f51d4a5a2955-maas-api-tls\") pod \"maas-api-e2e-ipp-a-626801-6f787498ff-hq648\" (UID: \"2889f9a4-a913-482c-b59d-f51d4a5a2955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:34:00.980374 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:00.980342    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-849cn\" (UniqueName: \"kubernetes.io/projected/2889f9a4-a913-482c-b59d-f51d4a5a2955-kube-api-access-849cn\") pod \"maas-api-e2e-ipp-a-626801-6f787498ff-hq648\" (UID: \"2889f9a4-a913-482c-b59d-f51d4a5a2955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:34:00.980614 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:00.980512    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2889f9a4-a913-482c-b59d-f51d4a5a2955-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-626801-6f787498ff-hq648\" (UID: \"2889f9a4-a913-482c-b59d-f51d4a5a2955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:34:00.981083 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:00.981065    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2889f9a4-a913-482c-b59d-f51d4a5a2955-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-626801-6f787498ff-hq648\" (UID: \"2889f9a4-a913-482c-b59d-f51d4a5a2955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:34:00.982891 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:00.982872    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2889f9a4-a913-482c-b59d-f51d4a5a2955-maas-api-tls\") pod \"maas-api-e2e-ipp-a-626801-6f787498ff-hq648\" (UID: \"2889f9a4-a913-482c-b59d-f51d4a5a2955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:34:00.992412 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:00.992381    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-849cn\" (UniqueName: \"kubernetes.io/projected/2889f9a4-a913-482c-b59d-f51d4a5a2955-kube-api-access-849cn\") pod \"maas-api-e2e-ipp-a-626801-6f787498ff-hq648\" (UID: \"2889f9a4-a913-482c-b59d-f51d4a5a2955\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:34:01.020868 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:01.020836    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:34:01.171103 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:01.171071    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"]
Jul 27 22:34:01.173937 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:34:01.173909    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2889f9a4_a913_482c_b59d_f51d4a5a2955.slice/crio-0a637585dffd46029c3c16e1ed9591597d351ab39995178daa616b942b0f8d30 WatchSource:0}: Error finding container 0a637585dffd46029c3c16e1ed9591597d351ab39995178daa616b942b0f8d30: Status 404 returned error can't find the container with id 0a637585dffd46029c3c16e1ed9591597d351ab39995178daa616b942b0f8d30
Jul 27 22:34:01.175468 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:01.175448    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 22:34:01.382825 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:01.382779    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648" event={"ID":"2889f9a4-a913-482c-b59d-f51d4a5a2955","Type":"ContainerStarted","Data":"0a637585dffd46029c3c16e1ed9591597d351ab39995178daa616b942b0f8d30"}
Jul 27 22:34:02.391027 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:02.390984    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648" event={"ID":"2889f9a4-a913-482c-b59d-f51d4a5a2955","Type":"ContainerStarted","Data":"b6f305394f661043e955fe9cbe48169d0f516a7f4cf59494baaa2f1b3624e1b9"}
Jul 27 22:34:02.391480 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:02.391159    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:34:02.409298 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:02.409235    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648" podStartSLOduration=1.7993591119999999 podStartE2EDuration="2.409205484s" totalImagesPullingTime="609.846372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:34:02.407771708 +0000 UTC m=+2927.219119080" watchObservedRunningTime="2026-07-27 22:34:02.409205484 +0000 UTC m=+2927.220552855"
Jul 27 22:34:08.401694 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:08.401664    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:34:28.079414 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.079374    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"]
Jul 27 22:34:28.083112 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.083087    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:34:28.086138 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.086116    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-adca75-serving-cert\""
Jul 27 22:34:28.100502 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.100478    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"]
Jul 27 22:34:28.217392 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.217351    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05de6874-cf3e-41fc-9a1a-e7834339121b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk\" (UID: \"05de6874-cf3e-41fc-9a1a-e7834339121b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:34:28.217392 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.217389    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05de6874-cf3e-41fc-9a1a-e7834339121b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk\" (UID: \"05de6874-cf3e-41fc-9a1a-e7834339121b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:34:28.217628 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.217420    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq9wq\" (UniqueName: \"kubernetes.io/projected/05de6874-cf3e-41fc-9a1a-e7834339121b-kube-api-access-mq9wq\") pod \"maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk\" (UID: \"05de6874-cf3e-41fc-9a1a-e7834339121b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:34:28.318022 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.317990    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05de6874-cf3e-41fc-9a1a-e7834339121b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk\" (UID: \"05de6874-cf3e-41fc-9a1a-e7834339121b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:34:28.318196 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.318027    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05de6874-cf3e-41fc-9a1a-e7834339121b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk\" (UID: \"05de6874-cf3e-41fc-9a1a-e7834339121b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:34:28.318196 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.318049    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq9wq\" (UniqueName: \"kubernetes.io/projected/05de6874-cf3e-41fc-9a1a-e7834339121b-kube-api-access-mq9wq\") pod \"maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk\" (UID: \"05de6874-cf3e-41fc-9a1a-e7834339121b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:34:28.318848 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.318791    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05de6874-cf3e-41fc-9a1a-e7834339121b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk\" (UID: \"05de6874-cf3e-41fc-9a1a-e7834339121b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:34:28.320662 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.320641    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05de6874-cf3e-41fc-9a1a-e7834339121b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk\" (UID: \"05de6874-cf3e-41fc-9a1a-e7834339121b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:34:28.327551 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.327530    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq9wq\" (UniqueName: \"kubernetes.io/projected/05de6874-cf3e-41fc-9a1a-e7834339121b-kube-api-access-mq9wq\") pod \"maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk\" (UID: \"05de6874-cf3e-41fc-9a1a-e7834339121b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:34:28.394131 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.394083    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:34:28.553013 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:28.552986    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"]
Jul 27 22:34:28.555379 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:34:28.555353    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05de6874_cf3e_41fc_9a1a_e7834339121b.slice/crio-293b560634823af5857a05f57f3b3d8cb6fb14b66dea3fc53d73269031bdcdac WatchSource:0}: Error finding container 293b560634823af5857a05f57f3b3d8cb6fb14b66dea3fc53d73269031bdcdac: Status 404 returned error can't find the container with id 293b560634823af5857a05f57f3b3d8cb6fb14b66dea3fc53d73269031bdcdac
Jul 27 22:34:29.539308 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:29.539274    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk" event={"ID":"05de6874-cf3e-41fc-9a1a-e7834339121b","Type":"ContainerStarted","Data":"3f0ae6fb4870d62c9f50df6d475deef92a99f1732ca29e42c8010aca0920d3f5"}
Jul 27 22:34:29.539724 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:29.539323    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk" event={"ID":"05de6874-cf3e-41fc-9a1a-e7834339121b","Type":"ContainerStarted","Data":"293b560634823af5857a05f57f3b3d8cb6fb14b66dea3fc53d73269031bdcdac"}
Jul 27 22:34:29.539724 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:29.539355    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:34:29.560109 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:29.560059    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk" podStartSLOduration=1.224415674 podStartE2EDuration="1.560043212s" totalImagesPullingTime="335.627538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:34:29.558033421 +0000 UTC m=+2954.369380793" watchObservedRunningTime="2026-07-27 22:34:29.560043212 +0000 UTC m=+2954.371390584"
Jul 27 22:34:35.550144 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:35.550114    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:34:49.702650 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:34:49.702571    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-p8vvq_1bd637d6-21ac-438c-be0c-6f2e856aea55/payload-processing/0.log"
Jul 27 22:35:15.841805 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:15.841772    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:35:15.854778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:15.854753    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:35:45.123798 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:45.123762    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-p8vvq_1bd637d6-21ac-438c-be0c-6f2e856aea55/payload-processing/0.log"
Jul 27 22:35:49.421984 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:49.421756    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"]
Jul 27 22:35:49.426669 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:49.426645    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:35:49.431051 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:49.431011    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-1368f7-serving-cert\""
Jul 27 22:35:49.440996 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:49.440963    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"]
Jul 27 22:35:49.449979 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:49.449944    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca23a792-840d-49ab-8a84-1cde123dae2e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm\" (UID: \"ca23a792-840d-49ab-8a84-1cde123dae2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:35:49.450125 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:49.450057    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca23a792-840d-49ab-8a84-1cde123dae2e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm\" (UID: \"ca23a792-840d-49ab-8a84-1cde123dae2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:35:49.450125 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:49.450103    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctvz5\" (UniqueName: \"kubernetes.io/projected/ca23a792-840d-49ab-8a84-1cde123dae2e-kube-api-access-ctvz5\") pod \"maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm\" (UID: \"ca23a792-840d-49ab-8a84-1cde123dae2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:35:49.550580 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:49.550538    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca23a792-840d-49ab-8a84-1cde123dae2e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm\" (UID: \"ca23a792-840d-49ab-8a84-1cde123dae2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:35:49.550778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:49.550635    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca23a792-840d-49ab-8a84-1cde123dae2e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm\" (UID: \"ca23a792-840d-49ab-8a84-1cde123dae2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:35:49.550778 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:49.550677    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctvz5\" (UniqueName: \"kubernetes.io/projected/ca23a792-840d-49ab-8a84-1cde123dae2e-kube-api-access-ctvz5\") pod \"maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm\" (UID: \"ca23a792-840d-49ab-8a84-1cde123dae2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:35:49.550994 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:35:49.550783    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-serving-cert: secret "maas-api-e2e-ipp-cleanup-1368f7-serving-cert" not found
Jul 27 22:35:49.550994 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:35:49.550875    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca23a792-840d-49ab-8a84-1cde123dae2e-maas-api-tls podName:ca23a792-840d-49ab-8a84-1cde123dae2e nodeName:}" failed. No retries permitted until 2026-07-27 22:35:50.050851902 +0000 UTC m=+3034.862199264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca23a792-840d-49ab-8a84-1cde123dae2e-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm" (UID: "ca23a792-840d-49ab-8a84-1cde123dae2e") : secret "maas-api-e2e-ipp-cleanup-1368f7-serving-cert" not found
Jul 27 22:35:49.551539 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:49.551503    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca23a792-840d-49ab-8a84-1cde123dae2e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm\" (UID: \"ca23a792-840d-49ab-8a84-1cde123dae2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:35:49.563914 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:49.563879    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctvz5\" (UniqueName: \"kubernetes.io/projected/ca23a792-840d-49ab-8a84-1cde123dae2e-kube-api-access-ctvz5\") pod \"maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm\" (UID: \"ca23a792-840d-49ab-8a84-1cde123dae2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:35:50.054763 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.054729    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca23a792-840d-49ab-8a84-1cde123dae2e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm\" (UID: \"ca23a792-840d-49ab-8a84-1cde123dae2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:35:50.057349 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.057319    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca23a792-840d-49ab-8a84-1cde123dae2e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm\" (UID: \"ca23a792-840d-49ab-8a84-1cde123dae2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:35:50.185449 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.185408    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"]
Jul 27 22:35:50.189284 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.189263    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"
Jul 27 22:35:50.192483 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.192442    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-1368f7-dockercfg-vfxgm\""
Jul 27 22:35:50.192908 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.192892    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-1368f7\""
Jul 27 22:35:50.206512 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.206481    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"]
Jul 27 22:35:50.234369 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.234320    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"]
Jul 27 22:35:50.238549 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.238531    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"
Jul 27 22:35:50.256798 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.256739    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"]
Jul 27 22:35:50.256923 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.256900    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b85d94-fc6a-4393-999f-e3ac240e6e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p\" (UID: \"07b85d94-fc6a-4393-999f-e3ac240e6e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"
Jul 27 22:35:50.257083 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.257064    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d396fe-26b9-495c-a23a-e3553dff82d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv\" (UID: \"84d396fe-26b9-495c-a23a-e3553dff82d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"
Jul 27 22:35:50.257144 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.257091    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrbvw\" (UniqueName: \"kubernetes.io/projected/84d396fe-26b9-495c-a23a-e3553dff82d5-kube-api-access-vrbvw\") pod \"payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv\" (UID: \"84d396fe-26b9-495c-a23a-e3553dff82d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"
Jul 27 22:35:50.343171 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.343080    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:35:50.358450 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.358418    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vclz4\" (UniqueName: \"kubernetes.io/projected/07b85d94-fc6a-4393-999f-e3ac240e6e27-kube-api-access-vclz4\") pod \"payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p\" (UID: \"07b85d94-fc6a-4393-999f-e3ac240e6e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"
Jul 27 22:35:50.358572 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.358480    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b85d94-fc6a-4393-999f-e3ac240e6e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p\" (UID: \"07b85d94-fc6a-4393-999f-e3ac240e6e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"
Jul 27 22:35:50.358629 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.358595    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d396fe-26b9-495c-a23a-e3553dff82d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv\" (UID: \"84d396fe-26b9-495c-a23a-e3553dff82d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"
Jul 27 22:35:50.358679 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.358624    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrbvw\" (UniqueName: \"kubernetes.io/projected/84d396fe-26b9-495c-a23a-e3553dff82d5-kube-api-access-vrbvw\") pod \"payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv\" (UID: \"84d396fe-26b9-495c-a23a-e3553dff82d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"
Jul 27 22:35:50.359244 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.359203    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b85d94-fc6a-4393-999f-e3ac240e6e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p\" (UID: \"07b85d94-fc6a-4393-999f-e3ac240e6e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"
Jul 27 22:35:50.359368 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.359349    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d396fe-26b9-495c-a23a-e3553dff82d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv\" (UID: \"84d396fe-26b9-495c-a23a-e3553dff82d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"
Jul 27 22:35:50.374569 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.374533    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrbvw\" (UniqueName: \"kubernetes.io/projected/84d396fe-26b9-495c-a23a-e3553dff82d5-kube-api-access-vrbvw\") pod \"payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv\" (UID: \"84d396fe-26b9-495c-a23a-e3553dff82d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"
Jul 27 22:35:50.459746 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.459428    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vclz4\" (UniqueName: \"kubernetes.io/projected/07b85d94-fc6a-4393-999f-e3ac240e6e27-kube-api-access-vclz4\") pod \"payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p\" (UID: \"07b85d94-fc6a-4393-999f-e3ac240e6e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"
Jul 27 22:35:50.470532 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.470502    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vclz4\" (UniqueName: \"kubernetes.io/projected/07b85d94-fc6a-4393-999f-e3ac240e6e27-kube-api-access-vclz4\") pod \"payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p\" (UID: \"07b85d94-fc6a-4393-999f-e3ac240e6e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"
Jul 27 22:35:50.500257 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.500210    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"
Jul 27 22:35:50.513130 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.513103    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"]
Jul 27 22:35:50.514454 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:35:50.514429    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca23a792_840d_49ab_8a84_1cde123dae2e.slice/crio-747b281f897e12a6081253728c79794a4e5cd9826ca4000648750b51d335965a WatchSource:0}: Error finding container 747b281f897e12a6081253728c79794a4e5cd9826ca4000648750b51d335965a: Status 404 returned error can't find the container with id 747b281f897e12a6081253728c79794a4e5cd9826ca4000648750b51d335965a
Jul 27 22:35:50.548351 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.548309    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"
Jul 27 22:35:50.665604 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.665510    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"]
Jul 27 22:35:50.666800 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:35:50.666765    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b85d94_fc6a_4393_999f_e3ac240e6e27.slice/crio-0225c7b8a827ca81db16230d6d393223b2f526719431d4a166c72c895a604e6f WatchSource:0}: Error finding container 0225c7b8a827ca81db16230d6d393223b2f526719431d4a166c72c895a604e6f: Status 404 returned error can't find the container with id 0225c7b8a827ca81db16230d6d393223b2f526719431d4a166c72c895a604e6f
Jul 27 22:35:50.723083 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.723056    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"]
Jul 27 22:35:50.725759 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:35:50.725732    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d396fe_26b9_495c_a23a_e3553dff82d5.slice/crio-7910f34ea059bc9cb7db40bd565ecc187e5f0df63abbd08a526459fca7128bb2 WatchSource:0}: Error finding container 7910f34ea059bc9cb7db40bd565ecc187e5f0df63abbd08a526459fca7128bb2: Status 404 returned error can't find the container with id 7910f34ea059bc9cb7db40bd565ecc187e5f0df63abbd08a526459fca7128bb2
Jul 27 22:35:50.965742 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.965705    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv" event={"ID":"84d396fe-26b9-495c-a23a-e3553dff82d5","Type":"ContainerStarted","Data":"5645286f8b3ef59a3f27a7156cd85d9aadd7e38ea87e2b36a50201922e54ca15"}
Jul 27 22:35:50.965953 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.965767    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv" event={"ID":"84d396fe-26b9-495c-a23a-e3553dff82d5","Type":"ContainerStarted","Data":"7910f34ea059bc9cb7db40bd565ecc187e5f0df63abbd08a526459fca7128bb2"}
Jul 27 22:35:50.965953 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.965794    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"
Jul 27 22:35:50.967281 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.967254    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p" event={"ID":"07b85d94-fc6a-4393-999f-e3ac240e6e27","Type":"ContainerStarted","Data":"f736ca5c59d4a3106a5bdff8355688ba5ecd9baf327bab3861601191bd48d4df"}
Jul 27 22:35:50.967375 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.967308    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p" event={"ID":"07b85d94-fc6a-4393-999f-e3ac240e6e27","Type":"ContainerStarted","Data":"0225c7b8a827ca81db16230d6d393223b2f526719431d4a166c72c895a604e6f"}
Jul 27 22:35:50.967375 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.967332    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"
Jul 27 22:35:50.968441 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:50.968417    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm" event={"ID":"ca23a792-840d-49ab-8a84-1cde123dae2e","Type":"ContainerStarted","Data":"747b281f897e12a6081253728c79794a4e5cd9826ca4000648750b51d335965a"}
Jul 27 22:35:51.028846 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:51.028798    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv" podStartSLOduration=1.028776151 podStartE2EDuration="1.028776151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:35:50.998480557 +0000 UTC m=+3035.809827953" watchObservedRunningTime="2026-07-27 22:35:51.028776151 +0000 UTC m=+3035.840123524"
Jul 27 22:35:51.029590 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:51.029558    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p" podStartSLOduration=1.029545397 podStartE2EDuration="1.029545397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:35:51.026981983 +0000 UTC m=+3035.838329357" watchObservedRunningTime="2026-07-27 22:35:51.029545397 +0000 UTC m=+3035.840892768"
Jul 27 22:35:51.976134 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:51.976095    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm" event={"ID":"ca23a792-840d-49ab-8a84-1cde123dae2e","Type":"ContainerStarted","Data":"1cd193a762d80226dd04bb5ed318634fe8d407b168bd03b230755ef56973aaf3"}
Jul 27 22:35:51.976620 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:51.976572    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:35:51.993203 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:51.993150    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm" podStartSLOduration=2.5004322549999998 podStartE2EDuration="2.993132306s" totalImagesPullingTime="492.700051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:35:51.991931348 +0000 UTC m=+3036.803278722" watchObservedRunningTime="2026-07-27 22:35:51.993132306 +0000 UTC m=+3036.804479683"
Jul 27 22:35:57.988067 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:35:57.988036    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:36:01.978425 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:01.978387    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"
Jul 27 22:36:01.978861 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:01.978617    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"
Jul 27 22:36:15.384336 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.384299    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"]
Jul 27 22:36:15.384808 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.384649    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm" podUID="ca23a792-840d-49ab-8a84-1cde123dae2e" containerName="maas-api" containerID="cri-o://1cd193a762d80226dd04bb5ed318634fe8d407b168bd03b230755ef56973aaf3" gracePeriod=30
Jul 27 22:36:15.444958 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.444916    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"]
Jul 27 22:36:15.445213 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.445168    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv" podUID="84d396fe-26b9-495c-a23a-e3553dff82d5" containerName="payload-processing" containerID="cri-o://5645286f8b3ef59a3f27a7156cd85d9aadd7e38ea87e2b36a50201922e54ca15" gracePeriod=30
Jul 27 22:36:15.463939 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.463912    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"]
Jul 27 22:36:15.464316 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.464164    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p" podUID="07b85d94-fc6a-4393-999f-e3ac240e6e27" containerName="payload-pre-processing" containerID="cri-o://f736ca5c59d4a3106a5bdff8355688ba5ecd9baf327bab3861601191bd48d4df" gracePeriod=30
Jul 27 22:36:15.740202 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.740176    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"
Jul 27 22:36:15.806519 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.806494    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:36:15.811125 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.811092    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"
Jul 27 22:36:15.904888 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.904796    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vclz4\" (UniqueName: \"kubernetes.io/projected/07b85d94-fc6a-4393-999f-e3ac240e6e27-kube-api-access-vclz4\") pod \"07b85d94-fc6a-4393-999f-e3ac240e6e27\" (UID: \"07b85d94-fc6a-4393-999f-e3ac240e6e27\") "
Jul 27 22:36:15.904888 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.904840    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrbvw\" (UniqueName: \"kubernetes.io/projected/84d396fe-26b9-495c-a23a-e3553dff82d5-kube-api-access-vrbvw\") pod \"84d396fe-26b9-495c-a23a-e3553dff82d5\" (UID: \"84d396fe-26b9-495c-a23a-e3553dff82d5\") "
Jul 27 22:36:15.904888 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.904861    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca23a792-840d-49ab-8a84-1cde123dae2e-openshift-service-ca\") pod \"ca23a792-840d-49ab-8a84-1cde123dae2e\" (UID: \"ca23a792-840d-49ab-8a84-1cde123dae2e\") "
Jul 27 22:36:15.904888 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.904879    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b85d94-fc6a-4393-999f-e3ac240e6e27-plugins-config-volume\") pod \"07b85d94-fc6a-4393-999f-e3ac240e6e27\" (UID: \"07b85d94-fc6a-4393-999f-e3ac240e6e27\") "
Jul 27 22:36:15.905279 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.904941    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d396fe-26b9-495c-a23a-e3553dff82d5-plugins-config-volume\") pod \"84d396fe-26b9-495c-a23a-e3553dff82d5\" (UID: \"84d396fe-26b9-495c-a23a-e3553dff82d5\") "
Jul 27 22:36:15.905279 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.904975    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctvz5\" (UniqueName: \"kubernetes.io/projected/ca23a792-840d-49ab-8a84-1cde123dae2e-kube-api-access-ctvz5\") pod \"ca23a792-840d-49ab-8a84-1cde123dae2e\" (UID: \"ca23a792-840d-49ab-8a84-1cde123dae2e\") "
Jul 27 22:36:15.905279 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.905050    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca23a792-840d-49ab-8a84-1cde123dae2e-maas-api-tls\") pod \"ca23a792-840d-49ab-8a84-1cde123dae2e\" (UID: \"ca23a792-840d-49ab-8a84-1cde123dae2e\") "
Jul 27 22:36:15.905442 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.905202    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca23a792-840d-49ab-8a84-1cde123dae2e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca23a792-840d-49ab-8a84-1cde123dae2e" (UID: "ca23a792-840d-49ab-8a84-1cde123dae2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:36:15.905442 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.905324    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b85d94-fc6a-4393-999f-e3ac240e6e27-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07b85d94-fc6a-4393-999f-e3ac240e6e27" (UID: "07b85d94-fc6a-4393-999f-e3ac240e6e27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:36:15.905546 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.905527    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca23a792-840d-49ab-8a84-1cde123dae2e-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:36:15.905600 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.905558    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d396fe-26b9-495c-a23a-e3553dff82d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84d396fe-26b9-495c-a23a-e3553dff82d5" (UID: "84d396fe-26b9-495c-a23a-e3553dff82d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:36:15.907389 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.907361    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca23a792-840d-49ab-8a84-1cde123dae2e-kube-api-access-ctvz5" (OuterVolumeSpecName: "kube-api-access-ctvz5") pod "ca23a792-840d-49ab-8a84-1cde123dae2e" (UID: "ca23a792-840d-49ab-8a84-1cde123dae2e"). InnerVolumeSpecName "kube-api-access-ctvz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:36:15.907473 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.907414    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca23a792-840d-49ab-8a84-1cde123dae2e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca23a792-840d-49ab-8a84-1cde123dae2e" (UID: "ca23a792-840d-49ab-8a84-1cde123dae2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:36:15.907554 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.907539    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b85d94-fc6a-4393-999f-e3ac240e6e27-kube-api-access-vclz4" (OuterVolumeSpecName: "kube-api-access-vclz4") pod "07b85d94-fc6a-4393-999f-e3ac240e6e27" (UID: "07b85d94-fc6a-4393-999f-e3ac240e6e27"). InnerVolumeSpecName "kube-api-access-vclz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:36:15.907630 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:15.907609    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d396fe-26b9-495c-a23a-e3553dff82d5-kube-api-access-vrbvw" (OuterVolumeSpecName: "kube-api-access-vrbvw") pod "84d396fe-26b9-495c-a23a-e3553dff82d5" (UID: "84d396fe-26b9-495c-a23a-e3553dff82d5"). InnerVolumeSpecName "kube-api-access-vrbvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:36:16.006095 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.006059    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca23a792-840d-49ab-8a84-1cde123dae2e-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:36:16.006095 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.006090    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vclz4\" (UniqueName: \"kubernetes.io/projected/07b85d94-fc6a-4393-999f-e3ac240e6e27-kube-api-access-vclz4\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:36:16.006095 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.006100    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrbvw\" (UniqueName: \"kubernetes.io/projected/84d396fe-26b9-495c-a23a-e3553dff82d5-kube-api-access-vrbvw\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:36:16.006352 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.006110    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b85d94-fc6a-4393-999f-e3ac240e6e27-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:36:16.006352 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.006119    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d396fe-26b9-495c-a23a-e3553dff82d5-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:36:16.006352 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.006127    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctvz5\" (UniqueName: \"kubernetes.io/projected/ca23a792-840d-49ab-8a84-1cde123dae2e-kube-api-access-ctvz5\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:36:16.117852 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.117814    2549 generic.go:356] "Generic (PLEG): container finished" podID="07b85d94-fc6a-4393-999f-e3ac240e6e27" containerID="f736ca5c59d4a3106a5bdff8355688ba5ecd9baf327bab3861601191bd48d4df" exitCode=0
Jul 27 22:36:16.118041 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.117858    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"
Jul 27 22:36:16.118041 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.117908    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p" event={"ID":"07b85d94-fc6a-4393-999f-e3ac240e6e27","Type":"ContainerDied","Data":"f736ca5c59d4a3106a5bdff8355688ba5ecd9baf327bab3861601191bd48d4df"}
Jul 27 22:36:16.118041 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.117970    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p" event={"ID":"07b85d94-fc6a-4393-999f-e3ac240e6e27","Type":"ContainerDied","Data":"0225c7b8a827ca81db16230d6d393223b2f526719431d4a166c72c895a604e6f"}
Jul 27 22:36:16.118041 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.117999    2549 scope.go:117] "RemoveContainer" containerID="f736ca5c59d4a3106a5bdff8355688ba5ecd9baf327bab3861601191bd48d4df"
Jul 27 22:36:16.119274 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.119251    2549 generic.go:356] "Generic (PLEG): container finished" podID="ca23a792-840d-49ab-8a84-1cde123dae2e" containerID="1cd193a762d80226dd04bb5ed318634fe8d407b168bd03b230755ef56973aaf3" exitCode=0
Jul 27 22:36:16.119407 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.119289    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"
Jul 27 22:36:16.119407 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.119348    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm" event={"ID":"ca23a792-840d-49ab-8a84-1cde123dae2e","Type":"ContainerDied","Data":"1cd193a762d80226dd04bb5ed318634fe8d407b168bd03b230755ef56973aaf3"}
Jul 27 22:36:16.119596 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.119575    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm" event={"ID":"ca23a792-840d-49ab-8a84-1cde123dae2e","Type":"ContainerDied","Data":"747b281f897e12a6081253728c79794a4e5cd9826ca4000648750b51d335965a"}
Jul 27 22:36:16.120707 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.120680    2549 generic.go:356] "Generic (PLEG): container finished" podID="84d396fe-26b9-495c-a23a-e3553dff82d5" containerID="5645286f8b3ef59a3f27a7156cd85d9aadd7e38ea87e2b36a50201922e54ca15" exitCode=0
Jul 27 22:36:16.120812 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.120724    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"
Jul 27 22:36:16.120812 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.120769    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv" event={"ID":"84d396fe-26b9-495c-a23a-e3553dff82d5","Type":"ContainerDied","Data":"5645286f8b3ef59a3f27a7156cd85d9aadd7e38ea87e2b36a50201922e54ca15"}
Jul 27 22:36:16.121086 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.120811    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv" event={"ID":"84d396fe-26b9-495c-a23a-e3553dff82d5","Type":"ContainerDied","Data":"7910f34ea059bc9cb7db40bd565ecc187e5f0df63abbd08a526459fca7128bb2"}
Jul 27 22:36:16.135938 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.128331    2549 scope.go:117] "RemoveContainer" containerID="f736ca5c59d4a3106a5bdff8355688ba5ecd9baf327bab3861601191bd48d4df"
Jul 27 22:36:16.135938 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:36:16.131376    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f736ca5c59d4a3106a5bdff8355688ba5ecd9baf327bab3861601191bd48d4df\": container with ID starting with f736ca5c59d4a3106a5bdff8355688ba5ecd9baf327bab3861601191bd48d4df not found: ID does not exist" containerID="f736ca5c59d4a3106a5bdff8355688ba5ecd9baf327bab3861601191bd48d4df"
Jul 27 22:36:16.135938 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.131411    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f736ca5c59d4a3106a5bdff8355688ba5ecd9baf327bab3861601191bd48d4df"} err="failed to get container status \"f736ca5c59d4a3106a5bdff8355688ba5ecd9baf327bab3861601191bd48d4df\": rpc error: code = NotFound desc = could not find container \"f736ca5c59d4a3106a5bdff8355688ba5ecd9baf327bab3861601191bd48d4df\": container with ID starting with f736ca5c59d4a3106a5bdff8355688ba5ecd9baf327bab3861601191bd48d4df not found: ID does not exist"
Jul 27 22:36:16.135938 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.131438    2549 scope.go:117] "RemoveContainer" containerID="1cd193a762d80226dd04bb5ed318634fe8d407b168bd03b230755ef56973aaf3"
Jul 27 22:36:16.144308 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.144282    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"]
Jul 27 22:36:16.145122 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.145108    2549 scope.go:117] "RemoveContainer" containerID="1cd193a762d80226dd04bb5ed318634fe8d407b168bd03b230755ef56973aaf3"
Jul 27 22:36:16.145425 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:36:16.145396    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd193a762d80226dd04bb5ed318634fe8d407b168bd03b230755ef56973aaf3\": container with ID starting with 1cd193a762d80226dd04bb5ed318634fe8d407b168bd03b230755ef56973aaf3 not found: ID does not exist" containerID="1cd193a762d80226dd04bb5ed318634fe8d407b168bd03b230755ef56973aaf3"
Jul 27 22:36:16.145542 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.145455    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd193a762d80226dd04bb5ed318634fe8d407b168bd03b230755ef56973aaf3"} err="failed to get container status \"1cd193a762d80226dd04bb5ed318634fe8d407b168bd03b230755ef56973aaf3\": rpc error: code = NotFound desc = could not find container \"1cd193a762d80226dd04bb5ed318634fe8d407b168bd03b230755ef56973aaf3\": container with ID starting with 1cd193a762d80226dd04bb5ed318634fe8d407b168bd03b230755ef56973aaf3 not found: ID does not exist"
Jul 27 22:36:16.145542 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.145477    2549 scope.go:117] "RemoveContainer" containerID="5645286f8b3ef59a3f27a7156cd85d9aadd7e38ea87e2b36a50201922e54ca15"
Jul 27 22:36:16.148447 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.148424    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1368f7-bd5d5675b-9ks2p"]
Jul 27 22:36:16.153914 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.153898    2549 scope.go:117] "RemoveContainer" containerID="5645286f8b3ef59a3f27a7156cd85d9aadd7e38ea87e2b36a50201922e54ca15"
Jul 27 22:36:16.154149 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:36:16.154132    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5645286f8b3ef59a3f27a7156cd85d9aadd7e38ea87e2b36a50201922e54ca15\": container with ID starting with 5645286f8b3ef59a3f27a7156cd85d9aadd7e38ea87e2b36a50201922e54ca15 not found: ID does not exist" containerID="5645286f8b3ef59a3f27a7156cd85d9aadd7e38ea87e2b36a50201922e54ca15"
Jul 27 22:36:16.154204 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.154155    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5645286f8b3ef59a3f27a7156cd85d9aadd7e38ea87e2b36a50201922e54ca15"} err="failed to get container status \"5645286f8b3ef59a3f27a7156cd85d9aadd7e38ea87e2b36a50201922e54ca15\": rpc error: code = NotFound desc = could not find container \"5645286f8b3ef59a3f27a7156cd85d9aadd7e38ea87e2b36a50201922e54ca15\": container with ID starting with 5645286f8b3ef59a3f27a7156cd85d9aadd7e38ea87e2b36a50201922e54ca15 not found: ID does not exist"
Jul 27 22:36:16.160151 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.160099    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"]
Jul 27 22:36:16.165813 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.165791    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1368f7-7c94d87b86-fggvm"]
Jul 27 22:36:16.177939 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.177915    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"]
Jul 27 22:36:16.186683 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:16.186658    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1368f7-6cc5ff65d-nhmfv"]
Jul 27 22:36:17.701236 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:17.701185    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07b85d94-fc6a-4393-999f-e3ac240e6e27" path="/var/lib/kubelet/pods/07b85d94-fc6a-4393-999f-e3ac240e6e27/volumes"
Jul 27 22:36:17.701629 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:17.701577    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84d396fe-26b9-495c-a23a-e3553dff82d5" path="/var/lib/kubelet/pods/84d396fe-26b9-495c-a23a-e3553dff82d5/volumes"
Jul 27 22:36:17.701910 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:17.701896    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca23a792-840d-49ab-8a84-1cde123dae2e" path="/var/lib/kubelet/pods/ca23a792-840d-49ab-8a84-1cde123dae2e/volumes"
Jul 27 22:36:45.761015 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:45.760113    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"]
Jul 27 22:36:45.761015 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:45.760431    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648" podUID="2889f9a4-a913-482c-b59d-f51d4a5a2955" containerName="maas-api" containerID="cri-o://b6f305394f661043e955fe9cbe48169d0f516a7f4cf59494baaa2f1b3624e1b9" gracePeriod=30
Jul 27 22:36:46.024424 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.024361    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:36:46.081552 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.081523    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-849cn\" (UniqueName: \"kubernetes.io/projected/2889f9a4-a913-482c-b59d-f51d4a5a2955-kube-api-access-849cn\") pod \"2889f9a4-a913-482c-b59d-f51d4a5a2955\" (UID: \"2889f9a4-a913-482c-b59d-f51d4a5a2955\") "
Jul 27 22:36:46.081719 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.081647    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2889f9a4-a913-482c-b59d-f51d4a5a2955-openshift-service-ca\") pod \"2889f9a4-a913-482c-b59d-f51d4a5a2955\" (UID: \"2889f9a4-a913-482c-b59d-f51d4a5a2955\") "
Jul 27 22:36:46.081719 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.081677    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2889f9a4-a913-482c-b59d-f51d4a5a2955-maas-api-tls\") pod \"2889f9a4-a913-482c-b59d-f51d4a5a2955\" (UID: \"2889f9a4-a913-482c-b59d-f51d4a5a2955\") "
Jul 27 22:36:46.082005 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.081982    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2889f9a4-a913-482c-b59d-f51d4a5a2955-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2889f9a4-a913-482c-b59d-f51d4a5a2955" (UID: "2889f9a4-a913-482c-b59d-f51d4a5a2955"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:36:46.083988 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.083954    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2889f9a4-a913-482c-b59d-f51d4a5a2955-kube-api-access-849cn" (OuterVolumeSpecName: "kube-api-access-849cn") pod "2889f9a4-a913-482c-b59d-f51d4a5a2955" (UID: "2889f9a4-a913-482c-b59d-f51d4a5a2955"). InnerVolumeSpecName "kube-api-access-849cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:36:46.084273 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.084248    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2889f9a4-a913-482c-b59d-f51d4a5a2955-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2889f9a4-a913-482c-b59d-f51d4a5a2955" (UID: "2889f9a4-a913-482c-b59d-f51d4a5a2955"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:36:46.182568 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.182538    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2889f9a4-a913-482c-b59d-f51d4a5a2955-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:36:46.182568 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.182564    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2889f9a4-a913-482c-b59d-f51d4a5a2955-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:36:46.182568 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.182573    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-849cn\" (UniqueName: \"kubernetes.io/projected/2889f9a4-a913-482c-b59d-f51d4a5a2955-kube-api-access-849cn\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:36:46.284447 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.284348    2549 generic.go:356] "Generic (PLEG): container finished" podID="2889f9a4-a913-482c-b59d-f51d4a5a2955" containerID="b6f305394f661043e955fe9cbe48169d0f516a7f4cf59494baaa2f1b3624e1b9" exitCode=0
Jul 27 22:36:46.284447 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.284402    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"
Jul 27 22:36:46.284447 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.284409    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648" event={"ID":"2889f9a4-a913-482c-b59d-f51d4a5a2955","Type":"ContainerDied","Data":"b6f305394f661043e955fe9cbe48169d0f516a7f4cf59494baaa2f1b3624e1b9"}
Jul 27 22:36:46.284716 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.284464    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648" event={"ID":"2889f9a4-a913-482c-b59d-f51d4a5a2955","Type":"ContainerDied","Data":"0a637585dffd46029c3c16e1ed9591597d351ab39995178daa616b942b0f8d30"}
Jul 27 22:36:46.284716 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.284494    2549 scope.go:117] "RemoveContainer" containerID="b6f305394f661043e955fe9cbe48169d0f516a7f4cf59494baaa2f1b3624e1b9"
Jul 27 22:36:46.301638 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.301615    2549 scope.go:117] "RemoveContainer" containerID="b6f305394f661043e955fe9cbe48169d0f516a7f4cf59494baaa2f1b3624e1b9"
Jul 27 22:36:46.301947 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:36:46.301927    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f305394f661043e955fe9cbe48169d0f516a7f4cf59494baaa2f1b3624e1b9\": container with ID starting with b6f305394f661043e955fe9cbe48169d0f516a7f4cf59494baaa2f1b3624e1b9 not found: ID does not exist" containerID="b6f305394f661043e955fe9cbe48169d0f516a7f4cf59494baaa2f1b3624e1b9"
Jul 27 22:36:46.302025 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.301956    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f305394f661043e955fe9cbe48169d0f516a7f4cf59494baaa2f1b3624e1b9"} err="failed to get container status \"b6f305394f661043e955fe9cbe48169d0f516a7f4cf59494baaa2f1b3624e1b9\": rpc error: code = NotFound desc = could not find container \"b6f305394f661043e955fe9cbe48169d0f516a7f4cf59494baaa2f1b3624e1b9\": container with ID starting with b6f305394f661043e955fe9cbe48169d0f516a7f4cf59494baaa2f1b3624e1b9 not found: ID does not exist"
Jul 27 22:36:46.312900 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.312838    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"]
Jul 27 22:36:46.315987 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:46.315963    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-626801-6f787498ff-hq648"]
Jul 27 22:36:47.701635 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:36:47.701602    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2889f9a4-a913-482c-b59d-f51d4a5a2955" path="/var/lib/kubelet/pods/2889f9a4-a913-482c-b59d-f51d4a5a2955/volumes"
Jul 27 22:37:49.761391 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:49.761357    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"]
Jul 27 22:37:49.761977 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:49.761583    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk" podUID="05de6874-cf3e-41fc-9a1a-e7834339121b" containerName="maas-api" containerID="cri-o://3f0ae6fb4870d62c9f50df6d475deef92a99f1732ca29e42c8010aca0920d3f5" gracePeriod=30
Jul 27 22:37:50.032372 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.032341    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:37:50.168520 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.168490    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05de6874-cf3e-41fc-9a1a-e7834339121b-openshift-service-ca\") pod \"05de6874-cf3e-41fc-9a1a-e7834339121b\" (UID: \"05de6874-cf3e-41fc-9a1a-e7834339121b\") "
Jul 27 22:37:50.168681 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.168588    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq9wq\" (UniqueName: \"kubernetes.io/projected/05de6874-cf3e-41fc-9a1a-e7834339121b-kube-api-access-mq9wq\") pod \"05de6874-cf3e-41fc-9a1a-e7834339121b\" (UID: \"05de6874-cf3e-41fc-9a1a-e7834339121b\") "
Jul 27 22:37:50.168681 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.168647    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05de6874-cf3e-41fc-9a1a-e7834339121b-maas-api-tls\") pod \"05de6874-cf3e-41fc-9a1a-e7834339121b\" (UID: \"05de6874-cf3e-41fc-9a1a-e7834339121b\") "
Jul 27 22:37:50.168901 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.168874    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05de6874-cf3e-41fc-9a1a-e7834339121b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05de6874-cf3e-41fc-9a1a-e7834339121b" (UID: "05de6874-cf3e-41fc-9a1a-e7834339121b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:37:50.170819 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.170791    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05de6874-cf3e-41fc-9a1a-e7834339121b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05de6874-cf3e-41fc-9a1a-e7834339121b" (UID: "05de6874-cf3e-41fc-9a1a-e7834339121b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:37:50.170819 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.170800    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05de6874-cf3e-41fc-9a1a-e7834339121b-kube-api-access-mq9wq" (OuterVolumeSpecName: "kube-api-access-mq9wq") pod "05de6874-cf3e-41fc-9a1a-e7834339121b" (UID: "05de6874-cf3e-41fc-9a1a-e7834339121b"). InnerVolumeSpecName "kube-api-access-mq9wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:37:50.269672 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.269584    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq9wq\" (UniqueName: \"kubernetes.io/projected/05de6874-cf3e-41fc-9a1a-e7834339121b-kube-api-access-mq9wq\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:37:50.269672 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.269614    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05de6874-cf3e-41fc-9a1a-e7834339121b-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:37:50.269672 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.269623    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05de6874-cf3e-41fc-9a1a-e7834339121b-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:37:50.626368 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.626329    2549 generic.go:356] "Generic (PLEG): container finished" podID="05de6874-cf3e-41fc-9a1a-e7834339121b" containerID="3f0ae6fb4870d62c9f50df6d475deef92a99f1732ca29e42c8010aca0920d3f5" exitCode=0
Jul 27 22:37:50.626557 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.626377    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"
Jul 27 22:37:50.626557 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.626420    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk" event={"ID":"05de6874-cf3e-41fc-9a1a-e7834339121b","Type":"ContainerDied","Data":"3f0ae6fb4870d62c9f50df6d475deef92a99f1732ca29e42c8010aca0920d3f5"}
Jul 27 22:37:50.626557 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.626477    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk" event={"ID":"05de6874-cf3e-41fc-9a1a-e7834339121b","Type":"ContainerDied","Data":"293b560634823af5857a05f57f3b3d8cb6fb14b66dea3fc53d73269031bdcdac"}
Jul 27 22:37:50.626557 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.626499    2549 scope.go:117] "RemoveContainer" containerID="3f0ae6fb4870d62c9f50df6d475deef92a99f1732ca29e42c8010aca0920d3f5"
Jul 27 22:37:50.636319 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.636301    2549 scope.go:117] "RemoveContainer" containerID="3f0ae6fb4870d62c9f50df6d475deef92a99f1732ca29e42c8010aca0920d3f5"
Jul 27 22:37:50.636578 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:37:50.636552    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0ae6fb4870d62c9f50df6d475deef92a99f1732ca29e42c8010aca0920d3f5\": container with ID starting with 3f0ae6fb4870d62c9f50df6d475deef92a99f1732ca29e42c8010aca0920d3f5 not found: ID does not exist" containerID="3f0ae6fb4870d62c9f50df6d475deef92a99f1732ca29e42c8010aca0920d3f5"
Jul 27 22:37:50.636646 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.636586    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0ae6fb4870d62c9f50df6d475deef92a99f1732ca29e42c8010aca0920d3f5"} err="failed to get container status \"3f0ae6fb4870d62c9f50df6d475deef92a99f1732ca29e42c8010aca0920d3f5\": rpc error: code = NotFound desc = could not find container \"3f0ae6fb4870d62c9f50df6d475deef92a99f1732ca29e42c8010aca0920d3f5\": container with ID starting with 3f0ae6fb4870d62c9f50df6d475deef92a99f1732ca29e42c8010aca0920d3f5 not found: ID does not exist"
Jul 27 22:37:50.648269 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.648245    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"]
Jul 27 22:37:50.653514 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:50.653490    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-adca75-6944d6d96-ldsxk"]
Jul 27 22:37:51.700281 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:37:51.700246    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05de6874-cf3e-41fc-9a1a-e7834339121b" path="/var/lib/kubelet/pods/05de6874-cf3e-41fc-9a1a-e7834339121b/volumes"
Jul 27 22:38:11.985002 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:11.984966    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"]
Jul 27 22:38:11.987685 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:11.985196    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn" podUID="61991ffa-5178-4f4d-bf85-0dbdf3b58882" containerName="maas-api" containerID="cri-o://c8e9c1cf62fb135207e2dc543e9404cb4e38148d8a84d043041f2fae7ab675a9" gracePeriod=30
Jul 27 22:38:12.042393 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.042353    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"]
Jul 27 22:38:12.042673 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.042619    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22" podUID="6bb37e2f-b130-4748-9414-cddb0ce4d922" containerName="payload-processing" containerID="cri-o://1b91b10cd2d5cd00304710a195e07bc28648a1804a3ed833e6da46bc59f5c61a" gracePeriod=30
Jul 27 22:38:12.051765 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.051722    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"]
Jul 27 22:38:12.051982 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.051954    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm" podUID="05987501-fdc8-4a57-92a6-62c587d609f3" containerName="payload-pre-processing" containerID="cri-o://367b18aef66e5ab284f52360549a8ac71a34813db201255660c8b2d843424113" gracePeriod=30
Jul 27 22:38:12.328334 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.328305    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"
Jul 27 22:38:12.352696 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.352671    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:38:12.358135 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.358114    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"
Jul 27 22:38:12.365489 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.365466    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05987501-fdc8-4a57-92a6-62c587d609f3-plugins-config-volume\") pod \"05987501-fdc8-4a57-92a6-62c587d609f3\" (UID: \"05987501-fdc8-4a57-92a6-62c587d609f3\") "
Jul 27 22:38:12.365605 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.365500    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th65p\" (UniqueName: \"kubernetes.io/projected/05987501-fdc8-4a57-92a6-62c587d609f3-kube-api-access-th65p\") pod \"05987501-fdc8-4a57-92a6-62c587d609f3\" (UID: \"05987501-fdc8-4a57-92a6-62c587d609f3\") "
Jul 27 22:38:12.365605 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.365535    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jskds\" (UniqueName: \"kubernetes.io/projected/61991ffa-5178-4f4d-bf85-0dbdf3b58882-kube-api-access-jskds\") pod \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\" (UID: \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\") "
Jul 27 22:38:12.365605 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.365553    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61991ffa-5178-4f4d-bf85-0dbdf3b58882-maas-api-tls\") pod \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\" (UID: \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\") "
Jul 27 22:38:12.365605 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.365570    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61991ffa-5178-4f4d-bf85-0dbdf3b58882-openshift-service-ca\") pod \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\" (UID: \"61991ffa-5178-4f4d-bf85-0dbdf3b58882\") "
Jul 27 22:38:12.365991 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.365966    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05987501-fdc8-4a57-92a6-62c587d609f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05987501-fdc8-4a57-92a6-62c587d609f3" (UID: "05987501-fdc8-4a57-92a6-62c587d609f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:38:12.366055 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.366017    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61991ffa-5178-4f4d-bf85-0dbdf3b58882-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "61991ffa-5178-4f4d-bf85-0dbdf3b58882" (UID: "61991ffa-5178-4f4d-bf85-0dbdf3b58882"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:38:12.367921 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.367901    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61991ffa-5178-4f4d-bf85-0dbdf3b58882-kube-api-access-jskds" (OuterVolumeSpecName: "kube-api-access-jskds") pod "61991ffa-5178-4f4d-bf85-0dbdf3b58882" (UID: "61991ffa-5178-4f4d-bf85-0dbdf3b58882"). InnerVolumeSpecName "kube-api-access-jskds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:38:12.368340 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.368315    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61991ffa-5178-4f4d-bf85-0dbdf3b58882-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "61991ffa-5178-4f4d-bf85-0dbdf3b58882" (UID: "61991ffa-5178-4f4d-bf85-0dbdf3b58882"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:38:12.368340 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.368323    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05987501-fdc8-4a57-92a6-62c587d609f3-kube-api-access-th65p" (OuterVolumeSpecName: "kube-api-access-th65p") pod "05987501-fdc8-4a57-92a6-62c587d609f3" (UID: "05987501-fdc8-4a57-92a6-62c587d609f3"). InnerVolumeSpecName "kube-api-access-th65p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:38:12.466634 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.466601    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jg2l\" (UniqueName: \"kubernetes.io/projected/6bb37e2f-b130-4748-9414-cddb0ce4d922-kube-api-access-7jg2l\") pod \"6bb37e2f-b130-4748-9414-cddb0ce4d922\" (UID: \"6bb37e2f-b130-4748-9414-cddb0ce4d922\") "
Jul 27 22:38:12.466820 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.466653    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb37e2f-b130-4748-9414-cddb0ce4d922-plugins-config-volume\") pod \"6bb37e2f-b130-4748-9414-cddb0ce4d922\" (UID: \"6bb37e2f-b130-4748-9414-cddb0ce4d922\") "
Jul 27 22:38:12.466927 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.466910    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jskds\" (UniqueName: \"kubernetes.io/projected/61991ffa-5178-4f4d-bf85-0dbdf3b58882-kube-api-access-jskds\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:38:12.466980 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.466934    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61991ffa-5178-4f4d-bf85-0dbdf3b58882-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:38:12.466980 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.466950    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61991ffa-5178-4f4d-bf85-0dbdf3b58882-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:38:12.466980 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.466965    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05987501-fdc8-4a57-92a6-62c587d609f3-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:38:12.467102 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.466979    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th65p\" (UniqueName: \"kubernetes.io/projected/05987501-fdc8-4a57-92a6-62c587d609f3-kube-api-access-th65p\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:38:12.467102 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.467068    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb37e2f-b130-4748-9414-cddb0ce4d922-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6bb37e2f-b130-4748-9414-cddb0ce4d922" (UID: "6bb37e2f-b130-4748-9414-cddb0ce4d922"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:38:12.468810 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.468786    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb37e2f-b130-4748-9414-cddb0ce4d922-kube-api-access-7jg2l" (OuterVolumeSpecName: "kube-api-access-7jg2l") pod "6bb37e2f-b130-4748-9414-cddb0ce4d922" (UID: "6bb37e2f-b130-4748-9414-cddb0ce4d922"). InnerVolumeSpecName "kube-api-access-7jg2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:38:12.567884 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.567794    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb37e2f-b130-4748-9414-cddb0ce4d922-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:38:12.567884 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.567829    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jg2l\" (UniqueName: \"kubernetes.io/projected/6bb37e2f-b130-4748-9414-cddb0ce4d922-kube-api-access-7jg2l\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:38:12.741713 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.741673    2549 generic.go:356] "Generic (PLEG): container finished" podID="05987501-fdc8-4a57-92a6-62c587d609f3" containerID="367b18aef66e5ab284f52360549a8ac71a34813db201255660c8b2d843424113" exitCode=0
Jul 27 22:38:12.741868 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.741722    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"
Jul 27 22:38:12.741868 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.741756    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm" event={"ID":"05987501-fdc8-4a57-92a6-62c587d609f3","Type":"ContainerDied","Data":"367b18aef66e5ab284f52360549a8ac71a34813db201255660c8b2d843424113"}
Jul 27 22:38:12.741868 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.741808    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm" event={"ID":"05987501-fdc8-4a57-92a6-62c587d609f3","Type":"ContainerDied","Data":"c1a4529b1ecfb0ba8e6ba4cab2f532f45dc39417471ba8b8708c71277a2dff2c"}
Jul 27 22:38:12.741868 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.741831    2549 scope.go:117] "RemoveContainer" containerID="367b18aef66e5ab284f52360549a8ac71a34813db201255660c8b2d843424113"
Jul 27 22:38:12.743068 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.743047    2549 generic.go:356] "Generic (PLEG): container finished" podID="6bb37e2f-b130-4748-9414-cddb0ce4d922" containerID="1b91b10cd2d5cd00304710a195e07bc28648a1804a3ed833e6da46bc59f5c61a" exitCode=0
Jul 27 22:38:12.743154 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.743095    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"
Jul 27 22:38:12.743154 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.743114    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22" event={"ID":"6bb37e2f-b130-4748-9414-cddb0ce4d922","Type":"ContainerDied","Data":"1b91b10cd2d5cd00304710a195e07bc28648a1804a3ed833e6da46bc59f5c61a"}
Jul 27 22:38:12.743277 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.743166    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22" event={"ID":"6bb37e2f-b130-4748-9414-cddb0ce4d922","Type":"ContainerDied","Data":"58bd55aaf9f28e6afee01d4cd8a6bb97b14115a7bfe2c669e9ec0dfff97cf810"}
Jul 27 22:38:12.744496 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.744470    2549 generic.go:356] "Generic (PLEG): container finished" podID="61991ffa-5178-4f4d-bf85-0dbdf3b58882" containerID="c8e9c1cf62fb135207e2dc543e9404cb4e38148d8a84d043041f2fae7ab675a9" exitCode=0
Jul 27 22:38:12.744605 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.744511    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"
Jul 27 22:38:12.744605 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.744529    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn" event={"ID":"61991ffa-5178-4f4d-bf85-0dbdf3b58882","Type":"ContainerDied","Data":"c8e9c1cf62fb135207e2dc543e9404cb4e38148d8a84d043041f2fae7ab675a9"}
Jul 27 22:38:12.744605 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.744564    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn" event={"ID":"61991ffa-5178-4f4d-bf85-0dbdf3b58882","Type":"ContainerDied","Data":"aee9764053f8b25e7e088d04eb031bc24acf828a6c20803d5f1bd66771907068"}
Jul 27 22:38:12.760168 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.760148    2549 scope.go:117] "RemoveContainer" containerID="367b18aef66e5ab284f52360549a8ac71a34813db201255660c8b2d843424113"
Jul 27 22:38:12.760534 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:38:12.760511    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"367b18aef66e5ab284f52360549a8ac71a34813db201255660c8b2d843424113\": container with ID starting with 367b18aef66e5ab284f52360549a8ac71a34813db201255660c8b2d843424113 not found: ID does not exist" containerID="367b18aef66e5ab284f52360549a8ac71a34813db201255660c8b2d843424113"
Jul 27 22:38:12.760633 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.760543    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367b18aef66e5ab284f52360549a8ac71a34813db201255660c8b2d843424113"} err="failed to get container status \"367b18aef66e5ab284f52360549a8ac71a34813db201255660c8b2d843424113\": rpc error: code = NotFound desc = could not find container \"367b18aef66e5ab284f52360549a8ac71a34813db201255660c8b2d843424113\": container with ID starting with 367b18aef66e5ab284f52360549a8ac71a34813db201255660c8b2d843424113 not found: ID does not exist"
Jul 27 22:38:12.760633 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.760568    2549 scope.go:117] "RemoveContainer" containerID="1b91b10cd2d5cd00304710a195e07bc28648a1804a3ed833e6da46bc59f5c61a"
Jul 27 22:38:12.769082 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.769052    2549 scope.go:117] "RemoveContainer" containerID="1b91b10cd2d5cd00304710a195e07bc28648a1804a3ed833e6da46bc59f5c61a"
Jul 27 22:38:12.769388 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:38:12.769366    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b91b10cd2d5cd00304710a195e07bc28648a1804a3ed833e6da46bc59f5c61a\": container with ID starting with 1b91b10cd2d5cd00304710a195e07bc28648a1804a3ed833e6da46bc59f5c61a not found: ID does not exist" containerID="1b91b10cd2d5cd00304710a195e07bc28648a1804a3ed833e6da46bc59f5c61a"
Jul 27 22:38:12.769481 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.769393    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b91b10cd2d5cd00304710a195e07bc28648a1804a3ed833e6da46bc59f5c61a"} err="failed to get container status \"1b91b10cd2d5cd00304710a195e07bc28648a1804a3ed833e6da46bc59f5c61a\": rpc error: code = NotFound desc = could not find container \"1b91b10cd2d5cd00304710a195e07bc28648a1804a3ed833e6da46bc59f5c61a\": container with ID starting with 1b91b10cd2d5cd00304710a195e07bc28648a1804a3ed833e6da46bc59f5c61a not found: ID does not exist"
Jul 27 22:38:12.769481 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.769417    2549 scope.go:117] "RemoveContainer" containerID="c8e9c1cf62fb135207e2dc543e9404cb4e38148d8a84d043041f2fae7ab675a9"
Jul 27 22:38:12.774687 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.774645    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"]
Jul 27 22:38:12.776803 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.776780    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6138cc-b54dfbb44-nbgrm"]
Jul 27 22:38:12.778581 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.778560    2549 scope.go:117] "RemoveContainer" containerID="c8e9c1cf62fb135207e2dc543e9404cb4e38148d8a84d043041f2fae7ab675a9"
Jul 27 22:38:12.778871 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:38:12.778841    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e9c1cf62fb135207e2dc543e9404cb4e38148d8a84d043041f2fae7ab675a9\": container with ID starting with c8e9c1cf62fb135207e2dc543e9404cb4e38148d8a84d043041f2fae7ab675a9 not found: ID does not exist" containerID="c8e9c1cf62fb135207e2dc543e9404cb4e38148d8a84d043041f2fae7ab675a9"
Jul 27 22:38:12.778942 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.778870    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e9c1cf62fb135207e2dc543e9404cb4e38148d8a84d043041f2fae7ab675a9"} err="failed to get container status \"c8e9c1cf62fb135207e2dc543e9404cb4e38148d8a84d043041f2fae7ab675a9\": rpc error: code = NotFound desc = could not find container \"c8e9c1cf62fb135207e2dc543e9404cb4e38148d8a84d043041f2fae7ab675a9\": container with ID starting with c8e9c1cf62fb135207e2dc543e9404cb4e38148d8a84d043041f2fae7ab675a9 not found: ID does not exist"
Jul 27 22:38:12.789281 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.789255    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"]
Jul 27 22:38:12.793720 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.793696    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-6138cc-59bb7c6f8b-sph22"]
Jul 27 22:38:12.807909 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.807884    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"]
Jul 27 22:38:12.812470 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:12.812448    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6138cc-6cbb788696-wtspn"]
Jul 27 22:38:13.700877 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:13.700836    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05987501-fdc8-4a57-92a6-62c587d609f3" path="/var/lib/kubelet/pods/05987501-fdc8-4a57-92a6-62c587d609f3/volumes"
Jul 27 22:38:13.701442 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:13.701421    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61991ffa-5178-4f4d-bf85-0dbdf3b58882" path="/var/lib/kubelet/pods/61991ffa-5178-4f4d-bf85-0dbdf3b58882/volumes"
Jul 27 22:38:13.701824 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:13.701803    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bb37e2f-b130-4748-9414-cddb0ce4d922" path="/var/lib/kubelet/pods/6bb37e2f-b130-4748-9414-cddb0ce4d922/volumes"
Jul 27 22:38:34.081636 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.081600    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"]
Jul 27 22:38:34.082157 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.081806    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9" podUID="8b8c63b8-69b4-4ca7-ac93-f5ac4807d251" containerName="maas-api" containerID="cri-o://f7e3f5767ca1cfdf516ac5c86194cd6ac43acbf31a6b1dd42fc6db6a1d034d5f" gracePeriod=30
Jul 27 22:38:34.165794 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.165759    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"]
Jul 27 22:38:34.166053 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.166020    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw" podUID="a0488a54-696d-420c-b60d-c7a4f32de184" containerName="payload-processing" containerID="cri-o://32a49c91b9590573f53d43986b1e93e7cba2d161c91a7ea3c7715708c648dee8" gracePeriod=30
Jul 27 22:38:34.178400 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.178370    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"]
Jul 27 22:38:34.178659 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.178610    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq" podUID="10c18dcd-919f-43ea-979b-c32d70922ad0" containerName="payload-pre-processing" containerID="cri-o://791aaa45ea8f65111090ad80c7b918a48378813baa327fda8cd09832006e5582" gracePeriod=30
Jul 27 22:38:34.429376 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.429348    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:38:34.446570 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.446533    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-maas-api-tls\") pod \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\" (UID: \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\") "
Jul 27 22:38:34.446570 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.446573    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-openshift-service-ca\") pod \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\" (UID: \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\") "
Jul 27 22:38:34.446810 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.446646    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfbtv\" (UniqueName: \"kubernetes.io/projected/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-kube-api-access-vfbtv\") pod \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\" (UID: \"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251\") "
Jul 27 22:38:34.447052 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.447012    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b8c63b8-69b4-4ca7-ac93-f5ac4807d251" (UID: "8b8c63b8-69b4-4ca7-ac93-f5ac4807d251"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:38:34.450554 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.450527    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b8c63b8-69b4-4ca7-ac93-f5ac4807d251" (UID: "8b8c63b8-69b4-4ca7-ac93-f5ac4807d251"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:38:34.451841 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.451816    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-kube-api-access-vfbtv" (OuterVolumeSpecName: "kube-api-access-vfbtv") pod "8b8c63b8-69b4-4ca7-ac93-f5ac4807d251" (UID: "8b8c63b8-69b4-4ca7-ac93-f5ac4807d251"). InnerVolumeSpecName "kube-api-access-vfbtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:38:34.477573 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.477545    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"
Jul 27 22:38:34.480343 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.480326    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"
Jul 27 22:38:34.548264 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.548206    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f5bj\" (UniqueName: \"kubernetes.io/projected/10c18dcd-919f-43ea-979b-c32d70922ad0-kube-api-access-2f5bj\") pod \"10c18dcd-919f-43ea-979b-c32d70922ad0\" (UID: \"10c18dcd-919f-43ea-979b-c32d70922ad0\") "
Jul 27 22:38:34.548458 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.548323    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0488a54-696d-420c-b60d-c7a4f32de184-plugins-config-volume\") pod \"a0488a54-696d-420c-b60d-c7a4f32de184\" (UID: \"a0488a54-696d-420c-b60d-c7a4f32de184\") "
Jul 27 22:38:34.548458 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.548354    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c18dcd-919f-43ea-979b-c32d70922ad0-plugins-config-volume\") pod \"10c18dcd-919f-43ea-979b-c32d70922ad0\" (UID: \"10c18dcd-919f-43ea-979b-c32d70922ad0\") "
Jul 27 22:38:34.548458 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.548411    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfjhk\" (UniqueName: \"kubernetes.io/projected/a0488a54-696d-420c-b60d-c7a4f32de184-kube-api-access-zfjhk\") pod \"a0488a54-696d-420c-b60d-c7a4f32de184\" (UID: \"a0488a54-696d-420c-b60d-c7a4f32de184\") "
Jul 27 22:38:34.548712 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.548694    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-maas-api-tls\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:38:34.548785 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.548719    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-openshift-service-ca\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:38:34.548785 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.548733    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfbtv\" (UniqueName: \"kubernetes.io/projected/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251-kube-api-access-vfbtv\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:38:34.548785 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.548732    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0488a54-696d-420c-b60d-c7a4f32de184-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0488a54-696d-420c-b60d-c7a4f32de184" (UID: "a0488a54-696d-420c-b60d-c7a4f32de184"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:38:34.549101 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.548778    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c18dcd-919f-43ea-979b-c32d70922ad0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "10c18dcd-919f-43ea-979b-c32d70922ad0" (UID: "10c18dcd-919f-43ea-979b-c32d70922ad0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:38:34.550504 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.550477    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c18dcd-919f-43ea-979b-c32d70922ad0-kube-api-access-2f5bj" (OuterVolumeSpecName: "kube-api-access-2f5bj") pod "10c18dcd-919f-43ea-979b-c32d70922ad0" (UID: "10c18dcd-919f-43ea-979b-c32d70922ad0"). InnerVolumeSpecName "kube-api-access-2f5bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:38:34.550767 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.550739    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0488a54-696d-420c-b60d-c7a4f32de184-kube-api-access-zfjhk" (OuterVolumeSpecName: "kube-api-access-zfjhk") pod "a0488a54-696d-420c-b60d-c7a4f32de184" (UID: "a0488a54-696d-420c-b60d-c7a4f32de184"). InnerVolumeSpecName "kube-api-access-zfjhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:38:34.649354 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.649321    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0488a54-696d-420c-b60d-c7a4f32de184-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:38:34.649354 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.649349    2549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c18dcd-919f-43ea-979b-c32d70922ad0-plugins-config-volume\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:38:34.649569 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.649366    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfjhk\" (UniqueName: \"kubernetes.io/projected/a0488a54-696d-420c-b60d-c7a4f32de184-kube-api-access-zfjhk\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:38:34.649569 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.649376    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f5bj\" (UniqueName: \"kubernetes.io/projected/10c18dcd-919f-43ea-979b-c32d70922ad0-kube-api-access-2f5bj\") on node \"ip-10-0-138-124.ec2.internal\" DevicePath \"\""
Jul 27 22:38:34.860375 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.860335    2549 generic.go:356] "Generic (PLEG): container finished" podID="8b8c63b8-69b4-4ca7-ac93-f5ac4807d251" containerID="f7e3f5767ca1cfdf516ac5c86194cd6ac43acbf31a6b1dd42fc6db6a1d034d5f" exitCode=0
Jul 27 22:38:34.860560 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.860395    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"
Jul 27 22:38:34.860560 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.860421    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9" event={"ID":"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251","Type":"ContainerDied","Data":"f7e3f5767ca1cfdf516ac5c86194cd6ac43acbf31a6b1dd42fc6db6a1d034d5f"}
Jul 27 22:38:34.860560 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.860479    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9" event={"ID":"8b8c63b8-69b4-4ca7-ac93-f5ac4807d251","Type":"ContainerDied","Data":"b8d082a1a72ef91eace31775a76164c7be41e7a093352c29aaabd6a0ef52b94b"}
Jul 27 22:38:34.860560 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.860509    2549 scope.go:117] "RemoveContainer" containerID="f7e3f5767ca1cfdf516ac5c86194cd6ac43acbf31a6b1dd42fc6db6a1d034d5f"
Jul 27 22:38:34.861762 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.861744    2549 generic.go:356] "Generic (PLEG): container finished" podID="a0488a54-696d-420c-b60d-c7a4f32de184" containerID="32a49c91b9590573f53d43986b1e93e7cba2d161c91a7ea3c7715708c648dee8" exitCode=0
Jul 27 22:38:34.861840 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.861804    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"
Jul 27 22:38:34.861840 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.861829    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw" event={"ID":"a0488a54-696d-420c-b60d-c7a4f32de184","Type":"ContainerDied","Data":"32a49c91b9590573f53d43986b1e93e7cba2d161c91a7ea3c7715708c648dee8"}
Jul 27 22:38:34.861942 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.861867    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw" event={"ID":"a0488a54-696d-420c-b60d-c7a4f32de184","Type":"ContainerDied","Data":"9ad3e2b63642bc51eada845a5c82dd0c1bfef484db7ad18c372a2b1efb46f975"}
Jul 27 22:38:34.863302 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.863277    2549 generic.go:356] "Generic (PLEG): container finished" podID="10c18dcd-919f-43ea-979b-c32d70922ad0" containerID="791aaa45ea8f65111090ad80c7b918a48378813baa327fda8cd09832006e5582" exitCode=0
Jul 27 22:38:34.863417 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.863334    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq" event={"ID":"10c18dcd-919f-43ea-979b-c32d70922ad0","Type":"ContainerDied","Data":"791aaa45ea8f65111090ad80c7b918a48378813baa327fda8cd09832006e5582"}
Jul 27 22:38:34.863417 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.863344    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"
Jul 27 22:38:34.863417 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.863363    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq" event={"ID":"10c18dcd-919f-43ea-979b-c32d70922ad0","Type":"ContainerDied","Data":"777d212f635ee4657750b4f209047cb3f77e45db596e1e8efaf6bddd2cf7bd11"}
Jul 27 22:38:34.870517 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.870499    2549 scope.go:117] "RemoveContainer" containerID="f7e3f5767ca1cfdf516ac5c86194cd6ac43acbf31a6b1dd42fc6db6a1d034d5f"
Jul 27 22:38:34.870805 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:38:34.870778    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e3f5767ca1cfdf516ac5c86194cd6ac43acbf31a6b1dd42fc6db6a1d034d5f\": container with ID starting with f7e3f5767ca1cfdf516ac5c86194cd6ac43acbf31a6b1dd42fc6db6a1d034d5f not found: ID does not exist" containerID="f7e3f5767ca1cfdf516ac5c86194cd6ac43acbf31a6b1dd42fc6db6a1d034d5f"
Jul 27 22:38:34.870867 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.870816    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e3f5767ca1cfdf516ac5c86194cd6ac43acbf31a6b1dd42fc6db6a1d034d5f"} err="failed to get container status \"f7e3f5767ca1cfdf516ac5c86194cd6ac43acbf31a6b1dd42fc6db6a1d034d5f\": rpc error: code = NotFound desc = could not find container \"f7e3f5767ca1cfdf516ac5c86194cd6ac43acbf31a6b1dd42fc6db6a1d034d5f\": container with ID starting with f7e3f5767ca1cfdf516ac5c86194cd6ac43acbf31a6b1dd42fc6db6a1d034d5f not found: ID does not exist"
Jul 27 22:38:34.870867 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.870834    2549 scope.go:117] "RemoveContainer" containerID="32a49c91b9590573f53d43986b1e93e7cba2d161c91a7ea3c7715708c648dee8"
Jul 27 22:38:34.879836 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.879816    2549 scope.go:117] "RemoveContainer" containerID="32a49c91b9590573f53d43986b1e93e7cba2d161c91a7ea3c7715708c648dee8"
Jul 27 22:38:34.880062 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:38:34.880047    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a49c91b9590573f53d43986b1e93e7cba2d161c91a7ea3c7715708c648dee8\": container with ID starting with 32a49c91b9590573f53d43986b1e93e7cba2d161c91a7ea3c7715708c648dee8 not found: ID does not exist" containerID="32a49c91b9590573f53d43986b1e93e7cba2d161c91a7ea3c7715708c648dee8"
Jul 27 22:38:34.880112 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.880068    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a49c91b9590573f53d43986b1e93e7cba2d161c91a7ea3c7715708c648dee8"} err="failed to get container status \"32a49c91b9590573f53d43986b1e93e7cba2d161c91a7ea3c7715708c648dee8\": rpc error: code = NotFound desc = could not find container \"32a49c91b9590573f53d43986b1e93e7cba2d161c91a7ea3c7715708c648dee8\": container with ID starting with 32a49c91b9590573f53d43986b1e93e7cba2d161c91a7ea3c7715708c648dee8 not found: ID does not exist"
Jul 27 22:38:34.880112 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.880083    2549 scope.go:117] "RemoveContainer" containerID="791aaa45ea8f65111090ad80c7b918a48378813baa327fda8cd09832006e5582"
Jul 27 22:38:34.888373 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.888349    2549 scope.go:117] "RemoveContainer" containerID="791aaa45ea8f65111090ad80c7b918a48378813baa327fda8cd09832006e5582"
Jul 27 22:38:34.888670 ip-10-0-138-124 kubenswrapper[2549]: E0727 22:38:34.888647    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791aaa45ea8f65111090ad80c7b918a48378813baa327fda8cd09832006e5582\": container with ID starting with 791aaa45ea8f65111090ad80c7b918a48378813baa327fda8cd09832006e5582 not found: ID does not exist" containerID="791aaa45ea8f65111090ad80c7b918a48378813baa327fda8cd09832006e5582"
Jul 27 22:38:34.888718 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.888682    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791aaa45ea8f65111090ad80c7b918a48378813baa327fda8cd09832006e5582"} err="failed to get container status \"791aaa45ea8f65111090ad80c7b918a48378813baa327fda8cd09832006e5582\": rpc error: code = NotFound desc = could not find container \"791aaa45ea8f65111090ad80c7b918a48378813baa327fda8cd09832006e5582\": container with ID starting with 791aaa45ea8f65111090ad80c7b918a48378813baa327fda8cd09832006e5582 not found: ID does not exist"
Jul 27 22:38:34.888915 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.888901    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"]
Jul 27 22:38:34.895410 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.895388    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0c79f1-77b7bbf67-n92m9"]
Jul 27 22:38:34.916195 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.916109    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"]
Jul 27 22:38:34.919617 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.919587    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0c79f1-59fc7f7546-46nhw"]
Jul 27 22:38:34.931443 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.931415    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"]
Jul 27 22:38:34.935311 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:34.935285    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-0c79f1-7dccffd48d-dvtgq"]
Jul 27 22:38:35.700671 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:35.700637    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c18dcd-919f-43ea-979b-c32d70922ad0" path="/var/lib/kubelet/pods/10c18dcd-919f-43ea-979b-c32d70922ad0/volumes"
Jul 27 22:38:35.701051 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:35.701022    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b8c63b8-69b4-4ca7-ac93-f5ac4807d251" path="/var/lib/kubelet/pods/8b8c63b8-69b4-4ca7-ac93-f5ac4807d251/volumes"
Jul 27 22:38:35.701357 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:35.701343    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0488a54-696d-420c-b60d-c7a4f32de184" path="/var/lib/kubelet/pods/a0488a54-696d-420c-b60d-c7a4f32de184/volumes"
Jul 27 22:38:51.867786 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:51.867748    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-dx46t_d09d1f65-5192-4d4e-84de-20e715df224a/manager/0.log"
Jul 27 22:38:52.001159 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:52.001122    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-p8tmv_b701741e-22b5-4e43-ae3e-c90642f4a309/manager/0.log"
Jul 27 22:38:52.143195 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:52.143088    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8599cc5679-t4hzv_beb9e0cf-fb12-40e2-a171-4a492d8debaf/manager/0.log"
Jul 27 22:38:52.264124 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:52.264092    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6n7dt_15fcbdb9-b762-47ea-bf94-f7d3901f16cd/server/0.log"
Jul 27 22:38:52.397659 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:52.397564    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-5kvmh_bacacc03-d38a-4c8a-9194-d636e52b5215/manager/0.log"
Jul 27 22:38:52.787017 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:52.786932    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7_910b8b4c-ff67-44dc-8549-c10011badb60/copy-manifests/0.log"
Jul 27 22:38:52.805523 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:52.805480    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7_910b8b4c-ff67-44dc-8549-c10011badb60/manager/0.log"
Jul 27 22:38:53.156873 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:53.156835    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-689cf98ffb-th6lm_6d76d30a-f5e8-47fd-a43c-5f787e777136/maas-api/0.log"
Jul 27 22:38:53.272902 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:53.272872    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753160-lpqtd_94cf85e1-d268-4318-89f5-9b95af84405a/cleanup/0.log"
Jul 27 22:38:53.628897 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:53.628865    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-lbsdc_5986b3d3-e094-44be-a406-7651490ff78c/postgres/0.log"
Jul 27 22:38:55.308588 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:55.308556    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5qqsp_e36fa849-8364-4983-a9cf-5ff5fca2e211/manager/0.log"
Jul 27 22:38:56.462063 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:56.462030    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-gwhvs_35d1ae81-f38e-4696-b7d6-60045fea8c7b/discovery/0.log"
Jul 27 22:38:56.588106 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:56.588071    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5794464b4b-4jqss_b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1/kube-auth-proxy/0.log"
Jul 27 22:38:56.974977 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:56.974944    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-5tfmg_e6cb51f2-feec-4aeb-bb85-216d1e2208ea/payload-pre-processing/0.log"
Jul 27 22:38:57.102757 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:57.102718    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-p8vvq_1bd637d6-21ac-438c-be0c-6f2e856aea55/payload-processing/0.log"
Jul 27 22:38:57.579814 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:57.579772    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-c4dq9_93b987c8-fb07-4985-b4b6-724842c11813/main/0.log"
Jul 27 22:38:57.828264 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:57.828214    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-cjnwt_3b6a7cc1-8dc6-4ed7-b99a-1bf21bc40dff/main/0.log"
Jul 27 22:38:57.955697 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:38:57.955659    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgtvvr_385046ce-9465-470f-9d4b-b60954cff4ea/main/0.log"
Jul 27 22:39:06.759885 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:06.759851    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8tk6w_4d6dfdf3-9d9e-473f-b1fc-0bc33d14cba2/global-pull-secret-syncer/0.log"
Jul 27 22:39:06.956060 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:06.956028    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zl79r_67a63f7d-7ca2-4626-abcd-5138b6359eb2/konnectivity-agent/0.log"
Jul 27 22:39:07.013776 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:07.013692    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-124.ec2.internal_83a6c3c959838c7e0ad7324eda9fbb06/haproxy/0.log"
Jul 27 22:39:12.558925 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:12.558896    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5qqsp_e36fa849-8364-4983-a9cf-5ff5fca2e211/manager/0.log"
Jul 27 22:39:15.407906 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:15.407866    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rq46q_16c022ac-d280-47d5-a150-76485eaec3f9/node-exporter/0.log"
Jul 27 22:39:15.435169 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:15.435140    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rq46q_16c022ac-d280-47d5-a150-76485eaec3f9/kube-rbac-proxy/0.log"
Jul 27 22:39:15.461565 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:15.461540    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rq46q_16c022ac-d280-47d5-a150-76485eaec3f9/init-textfile/0.log"
Jul 27 22:39:17.277243 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277189    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"]
Jul 27 22:39:17.277613 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277567    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2889f9a4-a913-482c-b59d-f51d4a5a2955" containerName="maas-api"
Jul 27 22:39:17.277613 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277578    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2889f9a4-a913-482c-b59d-f51d4a5a2955" containerName="maas-api"
Jul 27 22:39:17.277613 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277587    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca23a792-840d-49ab-8a84-1cde123dae2e" containerName="maas-api"
Jul 27 22:39:17.277613 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277592    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca23a792-840d-49ab-8a84-1cde123dae2e" containerName="maas-api"
Jul 27 22:39:17.277613 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277601    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07b85d94-fc6a-4393-999f-e3ac240e6e27" containerName="payload-pre-processing"
Jul 27 22:39:17.277613 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277607    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="07b85d94-fc6a-4393-999f-e3ac240e6e27" containerName="payload-pre-processing"
Jul 27 22:39:17.277613 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277613    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84d396fe-26b9-495c-a23a-e3553dff82d5" containerName="payload-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277619    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="84d396fe-26b9-495c-a23a-e3553dff82d5" containerName="payload-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277629    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0488a54-696d-420c-b60d-c7a4f32de184" containerName="payload-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277635    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0488a54-696d-420c-b60d-c7a4f32de184" containerName="payload-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277645    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05de6874-cf3e-41fc-9a1a-e7834339121b" containerName="maas-api"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277650    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="05de6874-cf3e-41fc-9a1a-e7834339121b" containerName="maas-api"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277655    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c18dcd-919f-43ea-979b-c32d70922ad0" containerName="payload-pre-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277660    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c18dcd-919f-43ea-979b-c32d70922ad0" containerName="payload-pre-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277668    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b8c63b8-69b4-4ca7-ac93-f5ac4807d251" containerName="maas-api"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277673    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b8c63b8-69b4-4ca7-ac93-f5ac4807d251" containerName="maas-api"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277680    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61991ffa-5178-4f4d-bf85-0dbdf3b58882" containerName="maas-api"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277686    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="61991ffa-5178-4f4d-bf85-0dbdf3b58882" containerName="maas-api"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277696    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb37e2f-b130-4748-9414-cddb0ce4d922" containerName="payload-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277701    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb37e2f-b130-4748-9414-cddb0ce4d922" containerName="payload-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277709    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05987501-fdc8-4a57-92a6-62c587d609f3" containerName="payload-pre-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277714    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="05987501-fdc8-4a57-92a6-62c587d609f3" containerName="payload-pre-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277768    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bb37e2f-b130-4748-9414-cddb0ce4d922" containerName="payload-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277774    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="07b85d94-fc6a-4393-999f-e3ac240e6e27" containerName="payload-pre-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277782    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="84d396fe-26b9-495c-a23a-e3553dff82d5" containerName="payload-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277788    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="05987501-fdc8-4a57-92a6-62c587d609f3" containerName="payload-pre-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277794    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0488a54-696d-420c-b60d-c7a4f32de184" containerName="payload-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277799    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="2889f9a4-a913-482c-b59d-f51d4a5a2955" containerName="maas-api"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277803    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="05de6874-cf3e-41fc-9a1a-e7834339121b" containerName="maas-api"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277808    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="61991ffa-5178-4f4d-bf85-0dbdf3b58882" containerName="maas-api"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277813    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b8c63b8-69b4-4ca7-ac93-f5ac4807d251" containerName="maas-api"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277819    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c18dcd-919f-43ea-979b-c32d70922ad0" containerName="payload-pre-processing"
Jul 27 22:39:17.277837 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.277824    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca23a792-840d-49ab-8a84-1cde123dae2e" containerName="maas-api"
Jul 27 22:39:17.282018 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.282002    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.286895 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.286873    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhm4t\"/\"openshift-service-ca.crt\""
Jul 27 22:39:17.287909 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.287885    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dhm4t\"/\"default-dockercfg-v997v\""
Jul 27 22:39:17.288481 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.288125    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhm4t\"/\"kube-root-ca.crt\""
Jul 27 22:39:17.297273 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.297234    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"]
Jul 27 22:39:17.412739 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.412679    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-bf6dz_ebc50b72-b32f-42b2-b817-09d4ed57857b/networking-console-plugin/0.log"
Jul 27 22:39:17.423589 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.423560    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/955d07d4-6811-4162-b95d-96b6301d8809-proc\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.423686 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.423619    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/955d07d4-6811-4162-b95d-96b6301d8809-podres\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.423686 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.423675    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/955d07d4-6811-4162-b95d-96b6301d8809-lib-modules\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.423776 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.423699    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/955d07d4-6811-4162-b95d-96b6301d8809-sys\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.423776 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.423738    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcvgx\" (UniqueName: \"kubernetes.io/projected/955d07d4-6811-4162-b95d-96b6301d8809-kube-api-access-gcvgx\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.524456 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.524420    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/955d07d4-6811-4162-b95d-96b6301d8809-podres\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.524651 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.524579    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/955d07d4-6811-4162-b95d-96b6301d8809-lib-modules\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.524651 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.524604    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/955d07d4-6811-4162-b95d-96b6301d8809-podres\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.524651 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.524614    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/955d07d4-6811-4162-b95d-96b6301d8809-sys\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.524817 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.524663    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/955d07d4-6811-4162-b95d-96b6301d8809-sys\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.524817 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.524669    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcvgx\" (UniqueName: \"kubernetes.io/projected/955d07d4-6811-4162-b95d-96b6301d8809-kube-api-access-gcvgx\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.524817 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.524729    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/955d07d4-6811-4162-b95d-96b6301d8809-proc\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.524817 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.524772    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/955d07d4-6811-4162-b95d-96b6301d8809-lib-modules\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.524817 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.524814    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/955d07d4-6811-4162-b95d-96b6301d8809-proc\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.534301 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.534208    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcvgx\" (UniqueName: \"kubernetes.io/projected/955d07d4-6811-4162-b95d-96b6301d8809-kube-api-access-gcvgx\") pod \"perf-node-gather-daemonset-w8bz5\" (UID: \"955d07d4-6811-4162-b95d-96b6301d8809\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.591824 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.591759    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:17.732676 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:17.732643    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"]
Jul 27 22:39:17.735810 ip-10-0-138-124 kubenswrapper[2549]: W0727 22:39:17.735787    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod955d07d4_6811_4162_b95d_96b6301d8809.slice/crio-905e6d7f75a4191bc73b521a6d2ca73661017172ea4749a8266d876461e543b4 WatchSource:0}: Error finding container 905e6d7f75a4191bc73b521a6d2ca73661017172ea4749a8266d876461e543b4: Status 404 returned error can't find the container with id 905e6d7f75a4191bc73b521a6d2ca73661017172ea4749a8266d876461e543b4
Jul 27 22:39:18.072877 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:18.072773    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5" event={"ID":"955d07d4-6811-4162-b95d-96b6301d8809","Type":"ContainerStarted","Data":"afbeab511d81ca8115b71ed6cb3d228e232083328f8c5b8f39f3db2338f3658a"}
Jul 27 22:39:18.072877 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:18.072826    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5" event={"ID":"955d07d4-6811-4162-b95d-96b6301d8809","Type":"ContainerStarted","Data":"905e6d7f75a4191bc73b521a6d2ca73661017172ea4749a8266d876461e543b4"}
Jul 27 22:39:18.073109 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:18.072933    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:18.093041 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:18.092990    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5" podStartSLOduration=1.092973711 podStartE2EDuration="1.092973711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 22:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:39:18.092644242 +0000 UTC m=+3242.903991615" watchObservedRunningTime="2026-07-27 22:39:18.092973711 +0000 UTC m=+3242.904321084"
Jul 27 22:39:18.161191 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:18.161149    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/1.log"
Jul 27 22:39:18.170679 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:18.170647    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4sp48_b1662a7c-ddf5-4b76-b4cf-57c34f091232/console-operator/2.log"
Jul 27 22:39:18.728260 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:18.728191    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-cpdtd_c70e3eca-af15-4b11-ab52-71b75ea220a4/download-server/0.log"
Jul 27 22:39:19.333429 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:19.333388    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-9glvr_69b53e4c-dae3-4de9-b6a8-4de71c24edd6/volume-data-source-validator/0.log"
Jul 27 22:39:20.519355 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:20.519319    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dpv6j_2c5933c6-64fd-4d0d-b8f9-c32fd675d89d/dns/0.log"
Jul 27 22:39:20.541180 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:20.541149    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dpv6j_2c5933c6-64fd-4d0d-b8f9-c32fd675d89d/kube-rbac-proxy/0.log"
Jul 27 22:39:20.592589 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:20.592557    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fk2vd_519f6723-a7eb-4bbd-82db-e6664825df3c/dns-node-resolver/0.log"
Jul 27 22:39:21.225876 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:21.225844    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-75kmz_135bb451-1678-49a0-b5bb-33e63a8ac0c2/node-ca/0.log"
Jul 27 22:39:22.533092 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:22.533050    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-gwhvs_35d1ae81-f38e-4696-b7d6-60045fea8c7b/discovery/0.log"
Jul 27 22:39:22.553514 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:22.553488    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5794464b4b-4jqss_b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1/kube-auth-proxy/0.log"
Jul 27 22:39:22.713440 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:22.713405    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-5tfmg_e6cb51f2-feec-4aeb-bb85-216d1e2208ea/payload-pre-processing/0.log"
Jul 27 22:39:22.755286 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:22.755259    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-p8vvq_1bd637d6-21ac-438c-be0c-6f2e856aea55/payload-processing/0.log"
Jul 27 22:39:23.626918 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:23.626887    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bfc7z_f2576233-1733-4b8a-9056-09acb732b3d0/serve-healthcheck-canary/0.log"
Jul 27 22:39:24.088144 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:24.088073    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-w8bz5"
Jul 27 22:39:24.398082 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:24.398056    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x7lth_dea6eb9c-7849-4fe0-9a80-d6f867b98ea1/kube-rbac-proxy/0.log"
Jul 27 22:39:24.421719 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:24.421688    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x7lth_dea6eb9c-7849-4fe0-9a80-d6f867b98ea1/exporter/0.log"
Jul 27 22:39:24.446205 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:24.446177    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x7lth_dea6eb9c-7849-4fe0-9a80-d6f867b98ea1/extractor/0.log"
Jul 27 22:39:26.966283 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:26.966217    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-dx46t_d09d1f65-5192-4d4e-84de-20e715df224a/manager/0.log"
Jul 27 22:39:27.356666 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:27.356627    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-p8tmv_b701741e-22b5-4e43-ae3e-c90642f4a309/manager/0.log"
Jul 27 22:39:27.587019 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:27.586981    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8599cc5679-t4hzv_beb9e0cf-fb12-40e2-a171-4a492d8debaf/manager/0.log"
Jul 27 22:39:27.610292 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:27.610193    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6n7dt_15fcbdb9-b762-47ea-bf94-f7d3901f16cd/server/0.log"
Jul 27 22:39:27.737110 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:27.737076    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-5kvmh_bacacc03-d38a-4c8a-9194-d636e52b5215/manager/0.log"
Jul 27 22:39:28.023350 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:28.023262    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7_910b8b4c-ff67-44dc-8549-c10011badb60/manager/0.log"
Jul 27 22:39:28.042497 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:28.042474    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6d4bf4d4-zrqc7_910b8b4c-ff67-44dc-8549-c10011badb60/copy-manifests/0.log"
Jul 27 22:39:28.617332 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:28.617286    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-gwhvs_35d1ae81-f38e-4696-b7d6-60045fea8c7b/discovery/0.log"
Jul 27 22:39:28.628185 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:28.628162    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5794464b4b-4jqss_b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1/kube-auth-proxy/0.log"
Jul 27 22:39:28.769939 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:28.769905    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-5tfmg_e6cb51f2-feec-4aeb-bb85-216d1e2208ea/payload-pre-processing/0.log"
Jul 27 22:39:28.800328 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:28.800292    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-p8vvq_1bd637d6-21ac-438c-be0c-6f2e856aea55/payload-processing/0.log"
Jul 27 22:39:29.480899 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:29.480867    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-j6sf2_0c29f149-e54a-41b5-accb-f1a848e0e2aa/openshift-lws-operator/0.log"
Jul 27 22:39:30.516099 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:30.516063    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-gwhvs_35d1ae81-f38e-4696-b7d6-60045fea8c7b/discovery/0.log"
Jul 27 22:39:30.527928 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:30.527900    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5794464b4b-4jqss_b0fe3c49-3ec7-4362-9a66-ec3f2d1fa6f1/kube-auth-proxy/0.log"
Jul 27 22:39:30.658247 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:30.658192    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-5tfmg_e6cb51f2-feec-4aeb-bb85-216d1e2208ea/payload-pre-processing/0.log"
Jul 27 22:39:30.688657 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:30.688623    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-p8vvq_1bd637d6-21ac-438c-be0c-6f2e856aea55/payload-processing/0.log"
Jul 27 22:39:35.176682 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:35.176640    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc_5c7b3d47-576c-4657-a558-69c78baeebad/kube-storage-version-migrator-operator/1.log"
Jul 27 22:39:35.178839 ip-10-0-138-124 kubenswrapper[2549]: I0727 22:39:35.178815    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-wwxhc_5c7b3d47-576c-4657-a558-69c78baeebad/kube-storage-version-migrator-operator/0.log"