Jul 24 18:38:14.592191 ip-10-0-128-111 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 24 18:38:14.592206 ip-10-0-128-111 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 24 18:38:14.592216 ip-10-0-128-111 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 24 18:38:14.592576 ip-10-0-128-111 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 24 18:38:24.841430 ip-10-0-128-111 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 24 18:38:24.841446 ip-10-0-128-111 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d0f503de9a7048e0b86ca8d46b3b07fa --
Jul 24 18:39:59.106935 ip-10-0-128-111 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 18:39:59.548314 ip-10-0-128-111 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 18:39:59.548314 ip-10-0-128-111 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 18:39:59.548314 ip-10-0-128-111 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 18:39:59.548314 ip-10-0-128-111 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 18:39:59.548314 ip-10-0-128-111 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 18:39:59.549332 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.549248    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 18:39:59.554959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.554939    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 18:39:59.554959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.554954    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 18:39:59.554959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.554960    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 18:39:59.554959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.554965    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.554970    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.554974    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.554979    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.554984    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.554988    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.554992    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.554996    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555000    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555004    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555008    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555013    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555017    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555022    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555026    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555031    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555037    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555043    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555047    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 18:39:59.555199 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555052    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555057    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555070    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555074    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555078    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555082    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555087    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555091    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555096    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555100    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555105    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555109    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555113    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555117    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555121    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555126    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555131    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555136    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555141    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555146    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 18:39:59.555870 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555150    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555154    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555158    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555163    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555167    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555171    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555176    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555181    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555185    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555189    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555194    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555197    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555204    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555210    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555214    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555218    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555255    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555261    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555267    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 18:39:59.556367 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555274    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555279    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555284    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555290    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555294    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555298    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555303    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555307    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555312    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555317    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555321    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555325    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555329    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555333    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555338    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555342    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555347    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555351    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555356    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555360    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 18:39:59.556887 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555364    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555368    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555372    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555376    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555383    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555387    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555391    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555395    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555400    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555404    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555409    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555413    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555417    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555422    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555426    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555430    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555434    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.555438    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556313    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556326    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 18:39:59.557365 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556330    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556333    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556336    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556339    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556342    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556345    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556348    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556351    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556353    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556357    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556360    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556363    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556366    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556368    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556372    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556375    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556378    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556381    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556383    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556385    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 18:39:59.557875 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556388    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556391    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556393    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556396    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556398    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556401    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556403    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556406    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556408    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556411    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556414    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556418    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556421    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556423    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556426    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556428    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556431    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556433    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556436    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556438    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 18:39:59.558369 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556440    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556443    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556446    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556448    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556450    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556453    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556455    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556458    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556460    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556464    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556468    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556471    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556473    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556476    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556478    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556482    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556484    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556487    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556489    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 18:39:59.558877 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556492    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556494    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556498    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556501    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556518    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556523    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556527    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556531    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556534    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556537    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556540    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556544    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556547    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556549    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556552    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556555    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556557    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556560    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556562    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556565    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 18:39:59.559353 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556567    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556570    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556572    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556574    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556577    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556579    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556582    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556584    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556587    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556590    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556595    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556598    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556601    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556604    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556607    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556610    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556613    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556617    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556989    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556995    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 18:39:59.559944 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.556999    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557002    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557005    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557008    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557010    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557013    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557016    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557018    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557020    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557023    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557026    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557028    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557031    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557033    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557036    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557039    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557042    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557044    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557047    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557049    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 18:39:59.560439 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557052    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557054    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557057    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557059    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557062    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557064    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557067    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557070    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557073    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557076    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557082    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557085    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557090    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557093    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557095    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557098    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557100    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557103    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557106    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557108    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 18:39:59.560959 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557111    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557114    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557117    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557120    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557123    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557127    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557129    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557132    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557135    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557137    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557140    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557142    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557145    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557147    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557150    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557153    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557155    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557158    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557161    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 18:39:59.561438 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557163    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557166    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557169    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557173    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557178    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557181    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557184    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557186    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557189    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557191    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557194    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557196    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557199    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557201    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557204    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557206    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557210    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557214    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557216    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 18:39:59.561970 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557220    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557223    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557225    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557228    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557231    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557235    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557238    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557240    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557243    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557245    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557248    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557250    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557253    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557256    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557259    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557261    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557264    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557266    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557269    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557339    2580 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557346    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 18:39:59.562444 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557352    2580 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557356    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557363    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557366    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557370    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557374    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557377    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557380    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557384    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557387    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557390    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557393    2580 flags.go:64] FLAG: --cgroup-root=""
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557396    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557399    2580 flags.go:64] FLAG: --client-ca-file=""
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557402    2580 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557407    2580 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557411    2580 flags.go:64] FLAG: --cluster-domain=""
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557414    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557417    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557421    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557424    2580 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557428    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557430    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 18:39:59.562968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557433    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557436    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557439    2580 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557442    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557445    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557448    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557451    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557455    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557458    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557461    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557464    2580 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557467    2580 flags.go:64] FLAG: --enable-server="true"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557470    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557474    2580 flags.go:64] FLAG: --event-burst="100"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557478    2580 flags.go:64] FLAG: --event-qps="50"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557480    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557483    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557486    2580 flags.go:64] FLAG: --eviction-hard=""
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557490    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557492    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557497    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557500    2580 flags.go:64] FLAG: --eviction-soft=""
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557502    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557505    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557510    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 18:39:59.563518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557513    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557516    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557519    2580 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557521    2580 flags.go:64] FLAG: --feature-gates=""
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557525    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557528    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557531    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557534    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557537    2580 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557540    2580 flags.go:64] FLAG: --help="false"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557543    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-128-111.ec2.internal"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557546    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557549    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557552    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557555    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557559    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557562    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557565    2580 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557567    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557570    2580 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557573    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557576    2580 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557579    2580 flags.go:64] FLAG: --kube-reserved=""
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557582    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 18:39:59.564169 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557584    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557587    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557590    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557594    2580 flags.go:64] FLAG: --lock-file=""
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557597    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557600    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557603    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557609    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557612    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557615    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557618    2580 flags.go:64] FLAG: --logging-format="text"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557620    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557624    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557626    2580 flags.go:64] FLAG: --manifest-url=""
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557629    2580 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557633    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557636    2580 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557653    2580 flags.go:64] FLAG: --max-pods="110"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557656    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557659    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557662    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557665    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557668    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557671    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557674    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 18:39:59.564779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557681    2580 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557684    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557687    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557690    2580 flags.go:64] FLAG: --pod-cidr=""
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557693    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557699    2580 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557702    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557705    2580 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557708    2580 flags.go:64] FLAG: --port="10250"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557711    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557715    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03ecaf9592bde4aa5"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557718    2580 flags.go:64] FLAG: --qos-reserved=""
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557722    2580 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557725    2580 flags.go:64] FLAG: --register-node="true"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557728    2580 flags.go:64] FLAG: --register-with-taints=""
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557732    2580 flags.go:64] FLAG: --registry-burst="10"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557735    2580 flags.go:64] FLAG: --registry-qps="5"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557738    2580 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557741    2580 flags.go:64] FLAG: --reserved-memory=""
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557744    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557747    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557750    2580 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557753    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557756    2580 flags.go:64] FLAG: --runonce="false"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557758    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 18:39:59.565418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557761    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557764    2580 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557767    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557770    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557773    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557776    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557779    2580 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557782    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557785    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557788    2580 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557791    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557794    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557796    2580 flags.go:64] FLAG: --system-cgroups=""
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557799    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557804    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557807    2580 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557810    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557815    2580 flags.go:64] FLAG: --tls-min-version=""
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557818    2580 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557821    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557824    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557826    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557831    2580 flags.go:64] FLAG: --v="2"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557835    2580 flags.go:64] FLAG: --version="false"
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557839    2580 flags.go:64] FLAG: --vmodule=""
Jul 24 18:39:59.566039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557842    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.557846    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557982    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557986    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557990    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557994    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.557998    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558001    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558004    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558007    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558009    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558012    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558014    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558017    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558020    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558024    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558028    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558030    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558034    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 18:39:59.566692 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558037    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558040    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558043    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558045    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558048    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558052    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558055    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558058    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558061    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558063    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558067    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558070    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558072    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558075    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558077    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558080    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558082    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558085    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558088    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558091    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 18:39:59.567162 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558094    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558096    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558099    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558101    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558104    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558106    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558109    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558111    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558114    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558117    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558120    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558122    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558125    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558127    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558130    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558132    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558135    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558138    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558141    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558143    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 18:39:59.567669 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558146    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558148    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558152    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558155    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558157    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558159    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558162    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558165    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558167    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558171    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558174    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558177    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558179    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558182    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558184    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558187    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558189    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558192    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558194    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 18:39:59.568161 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558197    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558199    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558202    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558210    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558213    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558216    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558218    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558220    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558223    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558226    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558229    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558232    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558234    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558237    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558239    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558243    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558245    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558248    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558251    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558253    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 18:39:59.568628 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558256    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 18:39:59.569127 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558258    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 18:39:59.569127 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.558261    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 18:39:59.569127 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.558266    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 18:39:59.569127 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.565386    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 18:39:59.569127 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.565401    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 18:39:59.569127 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.565419    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 18:39:59.569127 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.565425    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 18:39:59.569127 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565477    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 18:39:59.569127 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565482    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 18:39:59.569127 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565486    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 18:39:59.569127 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565489    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 18:39:59.569127 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565492    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 18:39:59.569127 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565495    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 18:39:59.569127 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565498    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565501    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565504    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565507    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565510    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565512    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565515    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565518    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565520    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565523    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565525    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565528    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565531    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565533    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565536    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565538    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565541    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565544    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565548    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565552    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 18:39:59.569463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565555    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565558    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565561    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565564    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565567    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565570    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565573    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565576    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565579    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565583    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565586    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565589    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565592    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565595    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565598    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565601    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565603    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565606    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565609    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 18:39:59.569966 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565611    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565614    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565616    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565619    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565621    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565624    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565626    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565629    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565632    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565634    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565637    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565657    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565662    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565665    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565667    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565670    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565673    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565675    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565678    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 18:39:59.570424 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565681    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565685    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565690    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565694    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565698    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565702    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565706    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565709    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565712    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565715    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565718    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565721    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565723    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565726    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565728    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565731    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565733    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565736    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565740    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565743    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 18:39:59.570919 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565747    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565750    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565752    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565755    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565758    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565760    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565763    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565765    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565768    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565770    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565773    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565776    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565778    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565781    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565783    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 18:39:59.571416 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.565788    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565893    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565898    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565901    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565904    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565906    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565910    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565912    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565915    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565918    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565920    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565923    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565925    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565928    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565930    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565932    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565935    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565938    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565940    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565943    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 18:39:59.571809 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565945    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565948    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565951    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565953    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565956    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565959    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565962    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565964    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565967    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565969    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565972    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565975    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565979    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565983    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565986    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565989    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565992    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565995    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.565997    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566000    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 18:39:59.572306 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566004    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566007    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566009    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566012    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566014    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566017    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566019    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566022    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566024    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566027    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566029    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566032    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566034    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566037    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566039    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566042    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566045    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566048    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566051    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566053    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 18:39:59.572804 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566056    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566059    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566062    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566064    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566068    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566072    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566074    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566077    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566079    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566082    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566084    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566087    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566089    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566092    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566095    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566097    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566100    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566102    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566105    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 18:39:59.573298 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566109    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566112    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566114    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566117    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566119    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566122    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566124    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566127    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566129    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566132    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566134    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566137    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566139    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566142    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566144    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566147    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566149    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566152    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566154    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 18:39:59.573774 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566156    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 18:39:59.574269 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:39:59.566159    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 18:39:59.574269 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.566164    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 18:39:59.574269 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.566255    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 18:39:59.574269 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.570366    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 18:39:59.574269 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.572019    2580 server.go:1019] "Starting client certificate rotation"
Jul 24 18:39:59.574269 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.572113    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 18:39:59.574269 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.572148    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 18:39:59.575003 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.574990    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 18:39:59.579361 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.579255    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 18:39:59.589709 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.589690    2580 log.go:25] "Validated CRI v1 runtime API"
Jul 24 18:39:59.592153 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.592139    2580 log.go:25] "Validated CRI v1 image API"
Jul 24 18:39:59.593010 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.592995    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 18:39:59.596571 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.596552    2580 fs.go:135] Filesystem UUIDs: map[022f0305-c8f4-4fcf-97af-952751de2a79:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e08e4564-6b27-43c2-aab6-5a90c0d63163:/dev/nvme0n1p4]
Jul 24 18:39:59.596629 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.596570    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 18:39:59.603300 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.603195    2580 manager.go:216] Machine: {Timestamp:2026-07-24 18:39:59.600854799 +0000 UTC m=+0.368251555 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098492 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2799bdcf9aaac24ba74e186b7831ef SystemUUID:ec2799bd-cf9a-aac2-4ba7-4e186b7831ef BootID:d0f503de-9a70-48e0-b86c-a8d46b3b07fa Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7e:0d:d7:d8:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7e:0d:d7:d8:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:f9:22:fc:7d:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 18:39:59.603300 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.603293    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 18:39:59.603426 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.603363    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 18:39:59.604724 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.604709    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 18:39:59.606711 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.606680    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 18:39:59.606837 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.606715    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-111.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 18:39:59.607547 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.607537    2580 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 18:39:59.607582 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.607549    2580 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 18:39:59.607582 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.607556    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 18:39:59.609322 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.609311    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 18:39:59.610753 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.610741    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 18:39:59.613626 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.613615    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 18:39:59.613759 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.613749    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 18:39:59.613802 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.613780    2580 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 18:39:59.613802 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.613798    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 18:39:59.613869 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.613812    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 18:39:59.613869 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.613821    2580 kubelet.go:398] "Adding apiserver pod source"
Jul 24 18:39:59.613869 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.613831    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 18:39:59.614179 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.614167    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 18:39:59.614228 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.614186    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 18:39:59.616432 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.616410    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 18:39:59.618954 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.618942    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 18:39:59.619011 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.618972    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 18:39:59.621100 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.621086    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 18:39:59.621175 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.621105    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 18:39:59.621175 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.621112    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 18:39:59.621175 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.621119    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 18:39:59.621175 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.621132    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 18:39:59.621175 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.621140    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 18:39:59.621175 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.621148    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 18:39:59.621175 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.621153    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 18:39:59.621175 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.621161    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 18:39:59.621175 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.621167    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 18:39:59.621175 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.621175    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 18:39:59.621429 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.621181    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 18:39:59.622090 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.622075    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 18:39:59.622130 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.622094    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 18:39:59.625274 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.625257    2580 server.go:1268] "Started kubelet"
Jul 24 18:39:59.625389 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.625366    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-111.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 18:39:59.625460 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.625414    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 18:39:59.625460 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.625415    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 18:39:59.625460 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.625426    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 18:39:59.627829 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.625608    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 18:39:59.627915 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.627861    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 18:39:59.628356 ip-10-0-128-111 systemd[1]: Started Kubernetes Kubelet.
Jul 24 18:39:59.630491 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.630375    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 18:39:59.631382 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.630906    2580 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 18:39:59.631883 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.631844    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 18:39:59.635621 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.634510    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-111.ec2.internal.18c54dd3dab8965d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-111.ec2.internal,UID:ip-10-0-128-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-111.ec2.internal,},FirstTimestamp:2026-07-24 18:39:59.625238109 +0000 UTC m=+0.392634867,LastTimestamp:2026-07-24 18:39:59.625238109 +0000 UTC m=+0.392634867,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-111.ec2.internal,}"
Jul 24 18:39:59.640629 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.640616    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 18:39:59.640699 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.640627    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 18:39:59.640756 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.640733    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:39:59.641466 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.641452    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 18:39:59.641563 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.641554    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 18:39:59.641670 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.641471    2580 factory.go:55] Registering systemd factory
Jul 24 18:39:59.641741 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.641679    2580 factory.go:223] Registration of the systemd container factory successfully
Jul 24 18:39:59.641741 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.641456    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 18:39:59.641741 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.641738    2580 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 18:39:59.641874 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.641747    2580 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 18:39:59.641916 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.641890    2580 factory.go:153] Registering CRI-O factory
Jul 24 18:39:59.641916 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.641907    2580 factory.go:223] Registration of the crio container factory successfully
Jul 24 18:39:59.642008 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.641958    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 18:39:59.642008 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.641988    2580 factory.go:103] Registering Raw factory
Jul 24 18:39:59.642008 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.642003    2580 manager.go:1196] Started watching for new ooms in manager
Jul 24 18:39:59.642377 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.642366    2580 manager.go:349] Starting recovery of all containers
Jul 24 18:39:59.644914 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.644881    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 18:39:59.645713 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.645138    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 18:39:59.652628 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.652614    2580 manager.go:354] Recovery completed
Jul 24 18:39:59.654091 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.654074    2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 24 18:39:59.656974 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.656858    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 18:39:59.658636 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.658614    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 18:39:59.658725 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.658662    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 18:39:59.658725 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.658673    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Jul 24 18:39:59.659239 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.659226    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 18:39:59.659239 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.659239    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 18:39:59.659369 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.659239    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 18:39:59.659369 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.659312    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 18:39:59.660035 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.660019    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 18:39:59.660092 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.660051    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 18:39:59.660092 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.660065    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Jul 24 18:39:59.660583 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.660510    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-111.ec2.internal.18c54dd3dcb62c19  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-111.ec2.internal,UID:ip-10-0-128-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-111.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-111.ec2.internal,},FirstTimestamp:2026-07-24 18:39:59.658634265 +0000 UTC m=+0.426031022,LastTimestamp:2026-07-24 18:39:59.658634265 +0000 UTC m=+0.426031022,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-111.ec2.internal,}"
Jul 24 18:39:59.661306 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.661293    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjlvh"
Jul 24 18:39:59.662115 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.662102    2580 policy_none.go:49] "None policy: Start"
Jul 24 18:39:59.662179 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.662120    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 18:39:59.662179 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.662133    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 18:39:59.665458 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.665443    2580 policy_none.go:47] "Start"
Jul 24 18:39:59.668535 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.668389    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjlvh"
Jul 24 18:39:59.671283 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.671208    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-111.ec2.internal.18c54dd3dcb6ae4a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-111.ec2.internal,UID:ip-10-0-128-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-111.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-111.ec2.internal,},FirstTimestamp:2026-07-24 18:39:59.658667594 +0000 UTC m=+0.426064351,LastTimestamp:2026-07-24 18:39:59.658667594 +0000 UTC m=+0.426064351,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-111.ec2.internal,}"
Jul 24 18:39:59.673060 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.673000    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-111.ec2.internal.18c54dd3dcb6d05b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-111.ec2.internal,UID:ip-10-0-128-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-111.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-111.ec2.internal,},FirstTimestamp:2026-07-24 18:39:59.658676315 +0000 UTC m=+0.426073072,LastTimestamp:2026-07-24 18:39:59.658676315 +0000 UTC m=+0.426073072,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-111.ec2.internal,}"
Jul 24 18:39:59.697198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.697181    2580 manager.go:337] "Starting Device Plugin manager"
Jul 24 18:39:59.697265 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.697211    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 18:39:59.697265 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.697221    2580 server.go:87] "Starting device plugin registration server"
Jul 24 18:39:59.697466 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.697451    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 18:39:59.697556 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.697466    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 18:39:59.697606 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.697581    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 18:39:59.697673 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.697665    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 18:39:59.697730 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.697674    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 18:39:59.698132 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.698112    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 18:39:59.698198 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.698159    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:39:59.714242 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.714222    2580 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2614 into the system container \"/system.slice\": <nil>"
Jul 24 18:39:59.797327 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.797293    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 18:39:59.799432 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.797595    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 18:39:59.799432 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.798518    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 18:39:59.799432 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.798552    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 18:39:59.799432 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.798554    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 18:39:59.799432 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.798566    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Jul 24 18:39:59.799432 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.798590    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-111.ec2.internal"
Jul 24 18:39:59.799432 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.798597    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 18:39:59.799432 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.798621    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 18:39:59.799432 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.798679    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 18:39:59.801182 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.801168    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 18:39:59.808499 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.808480    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-111.ec2.internal"
Jul 24 18:39:59.808574 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.808504    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-111.ec2.internal\": node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:39:59.843239 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.843106    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:39:59.899415 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.899395    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal"]
Jul 24 18:39:59.899488 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.899466    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 18:39:59.900225 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.900213    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 18:39:59.900301 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.900239    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 18:39:59.900301 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.900255    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Jul 24 18:39:59.902569 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.902557    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 18:39:59.902730 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.902714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Jul 24 18:39:59.902811 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.902744    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 18:39:59.903213 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.903193    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 18:39:59.903213 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.903214    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 18:39:59.903340 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.903241    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Jul 24 18:39:59.903340 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.903245    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 18:39:59.903340 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.903266    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 18:39:59.903340 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.903278    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Jul 24 18:39:59.905573 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.905557    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal"
Jul 24 18:39:59.905631 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.905592    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 18:39:59.906300 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.906283    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 18:39:59.906384 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.906311    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 18:39:59.906384 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.906324    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Jul 24 18:39:59.928566 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.928550    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-111.ec2.internal\" not found" node="ip-10-0-128-111.ec2.internal"
Jul 24 18:39:59.932651 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.932626    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-111.ec2.internal\" not found" node="ip-10-0-128-111.ec2.internal"
Jul 24 18:39:59.943843 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:39:59.943826    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:39:59.943922 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.943892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6550cfaced065c17322d923d252f38de-config\") pod \"kube-apiserver-proxy-ip-10-0-128-111.ec2.internal\" (UID: \"6550cfaced065c17322d923d252f38de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal"
Jul 24 18:39:59.943922 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.943911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05b93cd3ce9aed02e2f4eddde9bac7ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal\" (UID: \"05b93cd3ce9aed02e2f4eddde9bac7ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Jul 24 18:39:59.944002 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:39:59.943928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05b93cd3ce9aed02e2f4eddde9bac7ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal\" (UID: \"05b93cd3ce9aed02e2f4eddde9bac7ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Jul 24 18:40:00.044695 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:00.044664    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:40:00.044792 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:00.044701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05b93cd3ce9aed02e2f4eddde9bac7ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal\" (UID: \"05b93cd3ce9aed02e2f4eddde9bac7ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Jul 24 18:40:00.044792 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:00.044733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05b93cd3ce9aed02e2f4eddde9bac7ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal\" (UID: \"05b93cd3ce9aed02e2f4eddde9bac7ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Jul 24 18:40:00.044792 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:00.044756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6550cfaced065c17322d923d252f38de-config\") pod \"kube-apiserver-proxy-ip-10-0-128-111.ec2.internal\" (UID: \"6550cfaced065c17322d923d252f38de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal"
Jul 24 18:40:00.044792 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:00.044788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05b93cd3ce9aed02e2f4eddde9bac7ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal\" (UID: \"05b93cd3ce9aed02e2f4eddde9bac7ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Jul 24 18:40:00.044946 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:00.044800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05b93cd3ce9aed02e2f4eddde9bac7ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal\" (UID: \"05b93cd3ce9aed02e2f4eddde9bac7ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Jul 24 18:40:00.044946 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:00.044802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6550cfaced065c17322d923d252f38de-config\") pod \"kube-apiserver-proxy-ip-10-0-128-111.ec2.internal\" (UID: \"6550cfaced065c17322d923d252f38de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal"
Jul 24 18:40:00.145375 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:00.145320    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:40:00.231535 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:00.231506    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Jul 24 18:40:00.236501 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:00.236485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal"
Jul 24 18:40:00.245950 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:00.245934    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:40:00.346373 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:00.346352    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:40:01.838571 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:01.838524    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:40:01.916558 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:01.916531    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 18:40:01.939028 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:01.939004    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:40:01.963178 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:01.963160    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 18:40:01.963528 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:01.963269    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 18:40:01.963528 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:01.963297    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 18:40:02.039478 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:02.039457    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:40:02.068640 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:02.068613    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 18:35:01 +0000 UTC" deadline="2027-12-19 21:47:45.006194419 +0000 UTC"
Jul 24 18:40:02.068640 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:02.068636    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12315h7m42.937560599s"
Jul 24 18:40:02.140211 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:02.140156    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:40:02.215209 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:02.215190    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 18:40:02.216592 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:02.216573    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 18:40:02.216669 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:02.216604    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 18:40:02.216669 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:02.216617    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeHasSufficientPID"
Jul 24 18:40:02.241189 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:02.241168    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:40:02.341693 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:02.341665    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-111.ec2.internal\" not found"
Jul 24 18:40:02.354198 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:40:02.354162    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b93cd3ce9aed02e2f4eddde9bac7ee.slice/crio-67ca394a549cc8f766e7c233b5f100fd7d1b643156df6f3ab062a8d21649894a WatchSource:0}: Error finding container 67ca394a549cc8f766e7c233b5f100fd7d1b643156df6f3ab062a8d21649894a: Status 404 returned error can't find the container with id 67ca394a549cc8f766e7c233b5f100fd7d1b643156df6f3ab062a8d21649894a
Jul 24 18:40:02.360585 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:02.360565    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:40:02.403163 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:40:02.403138    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6550cfaced065c17322d923d252f38de.slice/crio-e74956b5b711e79f5ed58a24e2d7c21b00024e97ca17766e6ba87d5492bf2eef WatchSource:0}: Error finding container e74956b5b711e79f5ed58a24e2d7c21b00024e97ca17766e6ba87d5492bf2eef: Status 404 returned error can't find the container with id e74956b5b711e79f5ed58a24e2d7c21b00024e97ca17766e6ba87d5492bf2eef
Jul 24 18:40:02.413115 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:02.413098    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 18:40:02.433537 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:02.433515    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal"
Jul 24 18:40:02.445514 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:02.445492    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 18:40:02.446566 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:02.446552    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal"
Jul 24 18:40:02.455206 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:02.455192    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 18:40:02.538872 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:02.538849    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 18:40:03.007013 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.006962    2580 apiserver.go:52] "Watching apiserver"
Jul 24 18:40:03.012432 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.012409    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 18:40:03.012795 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.012773    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-vn9rh","openshift-dns/node-resolver-l9drv","openshift-image-registry/node-ca-9nzfl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal","openshift-multus/multus-additional-cni-plugins-9wq59","openshift-multus/network-metrics-daemon-xkxwb","openshift-network-operator/iptables-alerter-mjtg8","kube-system/konnectivity-agent-6mlfh","kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b","openshift-cluster-node-tuning-operator/tuned-mtfgj","openshift-multus/multus-pnpwv","openshift-network-diagnostics/network-check-target-f6w5m","openshift-ovn-kubernetes/ovnkube-node-hrxnr"]
Jul 24 18:40:03.015313 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.015291    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.017525 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.017499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:03.017632 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.017574    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:03.018002 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.017975    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 18:40:03.018502 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.018483    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 18:40:03.018598 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.018492    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 18:40:03.018924 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.018905    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6jdzp\""
Jul 24 18:40:03.019784 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.019762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9nzfl"
Jul 24 18:40:03.021395 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.021364    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 18:40:03.021941 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.021772    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 18:40:03.021941 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.021780    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wzgcf\""
Jul 24 18:40:03.022212 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.022191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.022825 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.022802    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 18:40:03.023990 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.023963    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 18:40:03.024286 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.024269    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 18:40:03.024412 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.024391    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kzx84\""
Jul 24 18:40:03.024518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.024423    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 18:40:03.024518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.024484    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 18:40:03.024518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.024487    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 18:40:03.024691 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.024677    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.025478 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.025463    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 18:40:03.026693 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.026676    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 18:40:03.026967 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.026794    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 18:40:03.027120 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.027068    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 18:40:03.027120 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.027090    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 18:40:03.027241 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.027169    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jlsnt\""
Jul 24 18:40:03.027294 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.027246    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 18:40:03.028573 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.028400    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:03.028573 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.028550    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:03.030465 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.030444    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6mlfh"
Jul 24 18:40:03.032525 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.032501    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 18:40:03.032650 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.032635    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 18:40:03.032793 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.032780    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 18:40:03.032938 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.032921    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lnnm4\""
Jul 24 18:40:03.033464 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.033447    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mjtg8"
Jul 24 18:40:03.035159 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.035138    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 18:40:03.035497 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.035423    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 18:40:03.035587 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.035524    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t9kcx\""
Jul 24 18:40:03.035587 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.035533    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 18:40:03.035739 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.035698    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l9drv"
Jul 24 18:40:03.037517 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.037465    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 18:40:03.037757 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.037729    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ndjxl\""
Jul 24 18:40:03.037881 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.037769    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 18:40:03.038311 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.038293    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:03.038426 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.038364    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:03.040481 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.040456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.042223 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.042200    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6mb7s\""
Jul 24 18:40:03.042331 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.042236    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 18:40:03.042331 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.042264    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 18:40:03.042703 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.042684    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.043958 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.043937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1455f583-c317-4edc-98d3-9c8a7f14d37b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.044054 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.043976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-systemd-units\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.044054 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-run-ovn\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.044162 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-node-log\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.044162 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e257688-dca4-4017-a362-6760fac66ee2-ovnkube-config\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.044162 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044094    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 18:40:03.044162 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-kubelet-dir\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.044162 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-registration-dir\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.044452 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-device-dir\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.044452 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1455f583-c317-4edc-98d3-9c8a7f14d37b-system-cni-dir\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.044452 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1455f583-c317-4edc-98d3-9c8a7f14d37b-os-release\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.044452 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1455f583-c317-4edc-98d3-9c8a7f14d37b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.044452 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-slash\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.044452 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-run-systemd\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.044452 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-kubelet\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.044452 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-socket-dir\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv5rx\" (UniqueName: \"kubernetes.io/projected/523ee06f-af02-40dd-8d8c-602f2f9a6452-kube-api-access-vv5rx\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-run-netns\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-cni-bin\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-cni-netd\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044567    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e257688-dca4-4017-a362-6760fac66ee2-env-overrides\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044602    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-etc-selinux\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044638    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6mkv9\""
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1455f583-c317-4edc-98d3-9c8a7f14d37b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48692fa5-2ce7-4575-bacb-e0e6c5ca40a1-agent-certs\") pod \"konnectivity-agent-6mlfh\" (UID: \"48692fa5-2ce7-4575-bacb-e0e6c5ca40a1\") " pod="kube-system/konnectivity-agent-6mlfh"
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-var-lib-openvswitch\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-run-openvswitch\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnqbv\" (UniqueName: \"kubernetes.io/projected/1455f583-c317-4edc-98d3-9c8a7f14d37b-kube-api-access-vnqbv\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73781c1b-297f-4e84-a295-c720ff6c9b6c-host-slash\") pod \"iptables-alerter-mjtg8\" (UID: \"73781c1b-297f-4e84-a295-c720ff6c9b6c\") " pod="openshift-network-operator/iptables-alerter-mjtg8"
Jul 24 18:40:03.044861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpkgb\" (UniqueName: \"kubernetes.io/projected/3e257688-dca4-4017-a362-6760fac66ee2-kube-api-access-fpkgb\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044881    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22cgz\" (UniqueName: \"kubernetes.io/projected/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-kube-api-access-22cgz\") pod \"network-metrics-daemon-xkxwb\" (UID: \"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0\") " pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28bdcc77-a763-49e4-abb3-98de9bd1d399-hosts-file\") pod \"node-resolver-l9drv\" (UID: \"28bdcc77-a763-49e4-abb3-98de9bd1d399\") " pod="openshift-dns/node-resolver-l9drv"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nknqv\" (UniqueName: \"kubernetes.io/projected/28bdcc77-a763-49e4-abb3-98de9bd1d399-kube-api-access-nknqv\") pod \"node-resolver-l9drv\" (UID: \"28bdcc77-a763-49e4-abb3-98de9bd1d399\") " pod="openshift-dns/node-resolver-l9drv"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.044976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-sys-fs\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2tng\" (UniqueName: \"kubernetes.io/projected/73781c1b-297f-4e84-a295-c720ff6c9b6c-kube-api-access-q2tng\") pod \"iptables-alerter-mjtg8\" (UID: \"73781c1b-297f-4e84-a295-c720ff6c9b6c\") " pod="openshift-network-operator/iptables-alerter-mjtg8"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-run-ovn-kubernetes\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74fqd\" (UniqueName: \"kubernetes.io/projected/2360ee4d-31bb-4de1-91ad-278352f88f22-kube-api-access-74fqd\") pod \"node-ca-9nzfl\" (UID: \"2360ee4d-31bb-4de1-91ad-278352f88f22\") " pod="openshift-image-registry/node-ca-9nzfl"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sb9t\" (UniqueName: \"kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t\") pod \"network-check-target-f6w5m\" (UID: \"d932da4a-af42-4d08-a40d-a1c2ce4e1123\") " pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045177    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73781c1b-297f-4e84-a295-c720ff6c9b6c-iptables-alerter-script\") pod \"iptables-alerter-mjtg8\" (UID: \"73781c1b-297f-4e84-a295-c720ff6c9b6c\") " pod="openshift-network-operator/iptables-alerter-mjtg8"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2360ee4d-31bb-4de1-91ad-278352f88f22-host\") pod \"node-ca-9nzfl\" (UID: \"2360ee4d-31bb-4de1-91ad-278352f88f22\") " pod="openshift-image-registry/node-ca-9nzfl"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2360ee4d-31bb-4de1-91ad-278352f88f22-serviceca\") pod \"node-ca-9nzfl\" (UID: \"2360ee4d-31bb-4de1-91ad-278352f88f22\") " pod="openshift-image-registry/node-ca-9nzfl"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-log-socket\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1455f583-c317-4edc-98d3-9c8a7f14d37b-cnibin\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1455f583-c317-4edc-98d3-9c8a7f14d37b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48692fa5-2ce7-4575-bacb-e0e6c5ca40a1-konnectivity-ca\") pod \"konnectivity-agent-6mlfh\" (UID: \"48692fa5-2ce7-4575-bacb-e0e6c5ca40a1\") " pod="kube-system/konnectivity-agent-6mlfh"
Jul 24 18:40:03.045572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-etc-openvswitch\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.046173 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e257688-dca4-4017-a362-6760fac66ee2-ovn-node-metrics-cert\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.046173 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e257688-dca4-4017-a362-6760fac66ee2-ovnkube-script-lib\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.046173 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs\") pod \"network-metrics-daemon-xkxwb\" (UID: \"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0\") " pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:03.046173 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.045664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28bdcc77-a763-49e4-abb3-98de9bd1d399-tmp-dir\") pod \"node-resolver-l9drv\" (UID: \"28bdcc77-a763-49e4-abb3-98de9bd1d399\") " pod="openshift-dns/node-resolver-l9drv"
Jul 24 18:40:03.073933 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.073914    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kcrn6"
Jul 24 18:40:03.081583 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.081561    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kcrn6"
Jul 24 18:40:03.135028 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.134857    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 18:40:03.146548 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bc4eada-1823-4bc2-add7-98ef884c80d4-kubelet-config\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:03.146745 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-var-lib-kubelet\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.146745 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-multus-socket-dir-parent\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.146745 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-run-ovn-kubernetes\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.146745 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-multus-conf-dir\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.146745 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2360ee4d-31bb-4de1-91ad-278352f88f22-serviceca\") pod \"node-ca-9nzfl\" (UID: \"2360ee4d-31bb-4de1-91ad-278352f88f22\") " pod="openshift-image-registry/node-ca-9nzfl"
Jul 24 18:40:03.146745 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-sysctl-d\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.146745 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-run-ovn-kubernetes\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.146745 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-systemd\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.147147 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-run\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.147147 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/403dce40-1510-40cb-a119-e5bba09ed289-tmp\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.147147 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-system-cni-dir\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.147147 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1455f583-c317-4edc-98d3-9c8a7f14d37b-cnibin\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.147147 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-etc-openvswitch\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.147147 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e257688-dca4-4017-a362-6760fac66ee2-ovnkube-script-lib\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.147147 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs\") pod \"network-metrics-daemon-xkxwb\" (UID: \"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0\") " pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:03.147147 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28bdcc77-a763-49e4-abb3-98de9bd1d399-tmp-dir\") pod \"node-resolver-l9drv\" (UID: \"28bdcc77-a763-49e4-abb3-98de9bd1d399\") " pod="openshift-dns/node-resolver-l9drv"
Jul 24 18:40:03.147147 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.146992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-systemd-units\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.147586 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2360ee4d-31bb-4de1-91ad-278352f88f22-serviceca\") pod \"node-ca-9nzfl\" (UID: \"2360ee4d-31bb-4de1-91ad-278352f88f22\") " pod="openshift-image-registry/node-ca-9nzfl"
Jul 24 18:40:03.147586 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147298    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28bdcc77-a763-49e4-abb3-98de9bd1d399-tmp-dir\") pod \"node-resolver-l9drv\" (UID: \"28bdcc77-a763-49e4-abb3-98de9bd1d399\") " pod="openshift-dns/node-resolver-l9drv"
Jul 24 18:40:03.147586 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-run-ovn\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.147586 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e257688-dca4-4017-a362-6760fac66ee2-ovnkube-config\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.147586 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:03.147586 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-systemd-units\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.147586 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.147486    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 18:40:03.147586 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/feb5428a-ec41-4135-9a40-5bff75da000b-cni-binary-copy\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.147586 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-run-k8s-cni-cncf-io\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.147586 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-etc-kubernetes\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28bdcc77-a763-49e4-abb3-98de9bd1d399-hosts-file\") pod \"node-resolver-l9drv\" (UID: \"28bdcc77-a763-49e4-abb3-98de9bd1d399\") " pod="openshift-dns/node-resolver-l9drv"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nknqv\" (UniqueName: \"kubernetes.io/projected/28bdcc77-a763-49e4-abb3-98de9bd1d399-kube-api-access-nknqv\") pod \"node-resolver-l9drv\" (UID: \"28bdcc77-a763-49e4-abb3-98de9bd1d399\") " pod="openshift-dns/node-resolver-l9drv"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-kubelet-dir\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-device-dir\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1455f583-c317-4edc-98d3-9c8a7f14d37b-system-cni-dir\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-slash\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-cni-bin\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-cni-netd\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-modprobe-d\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-kubelet\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-sysconfig\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-kubernetes\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-hostroot\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.147968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wprf\" (UniqueName: \"kubernetes.io/projected/feb5428a-ec41-4135-9a40-5bff75da000b-kube-api-access-7wprf\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48692fa5-2ce7-4575-bacb-e0e6c5ca40a1-agent-certs\") pod \"konnectivity-agent-6mlfh\" (UID: \"48692fa5-2ce7-4575-bacb-e0e6c5ca40a1\") " pod="kube-system/konnectivity-agent-6mlfh"
Jul 24 18:40:03.148065 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e257688-dca4-4017-a362-6760fac66ee2-ovnkube-config\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-run-openvswitch\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-cni-netd\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/403dce40-1510-40cb-a119-e5bba09ed289-etc-tuned\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-var-lib-kubelet\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148151    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-kubelet\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnqbv\" (UniqueName: \"kubernetes.io/projected/1455f583-c317-4edc-98d3-9c8a7f14d37b-kube-api-access-vnqbv\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1455f583-c317-4edc-98d3-9c8a7f14d37b-system-cni-dir\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bc4eada-1823-4bc2-add7-98ef884c80d4-dbus\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-sysctl-conf\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-slash\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-host\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-os-release\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-run-multus-certs\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-sys-fs\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2tng\" (UniqueName: \"kubernetes.io/projected/73781c1b-297f-4e84-a295-c720ff6c9b6c-kube-api-access-q2tng\") pod \"iptables-alerter-mjtg8\" (UID: \"73781c1b-297f-4e84-a295-c720ff6c9b6c\") " pod="openshift-network-operator/iptables-alerter-mjtg8"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1455f583-c317-4edc-98d3-9c8a7f14d37b-cnibin\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.148796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gn6g\" (UniqueName: \"kubernetes.io/projected/403dce40-1510-40cb-a119-e5bba09ed289-kube-api-access-9gn6g\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/feb5428a-ec41-4135-9a40-5bff75da000b-multus-daemon-config\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-kubelet-dir\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74fqd\" (UniqueName: \"kubernetes.io/projected/2360ee4d-31bb-4de1-91ad-278352f88f22-kube-api-access-74fqd\") pod \"node-ca-9nzfl\" (UID: \"2360ee4d-31bb-4de1-91ad-278352f88f22\") " pod="openshift-image-registry/node-ca-9nzfl"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148552    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb9t\" (UniqueName: \"kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t\") pod \"network-check-target-f6w5m\" (UID: \"d932da4a-af42-4d08-a40d-a1c2ce4e1123\") " pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73781c1b-297f-4e84-a295-c720ff6c9b6c-iptables-alerter-script\") pod \"iptables-alerter-mjtg8\" (UID: \"73781c1b-297f-4e84-a295-c720ff6c9b6c\") " pod="openshift-network-operator/iptables-alerter-mjtg8"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2360ee4d-31bb-4de1-91ad-278352f88f22-host\") pod \"node-ca-9nzfl\" (UID: \"2360ee4d-31bb-4de1-91ad-278352f88f22\") " pod="openshift-image-registry/node-ca-9nzfl"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-log-socket\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-device-dir\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1455f583-c317-4edc-98d3-9c8a7f14d37b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-etc-openvswitch\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48692fa5-2ce7-4575-bacb-e0e6c5ca40a1-konnectivity-ca\") pod \"konnectivity-agent-6mlfh\" (UID: \"48692fa5-2ce7-4575-bacb-e0e6c5ca40a1\") " pod="kube-system/konnectivity-agent-6mlfh"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e257688-dca4-4017-a362-6760fac66ee2-ovn-node-metrics-cert\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1455f583-c317-4edc-98d3-9c8a7f14d37b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-node-log\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-sys\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.149485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-multus-cni-dir\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-registration-dir\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1455f583-c317-4edc-98d3-9c8a7f14d37b-os-release\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1455f583-c317-4edc-98d3-9c8a7f14d37b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-run-systemd\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-var-lib-cni-bin\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-var-lib-cni-multus\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-sys-fs\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-socket-dir\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.148226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-run-ovn\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv5rx\" (UniqueName: \"kubernetes.io/projected/523ee06f-af02-40dd-8d8c-602f2f9a6452-kube-api-access-vv5rx\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-run-netns\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28bdcc77-a763-49e4-abb3-98de9bd1d399-hosts-file\") pod \"node-resolver-l9drv\" (UID: \"28bdcc77-a763-49e4-abb3-98de9bd1d399\") " pod="openshift-dns/node-resolver-l9drv"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e257688-dca4-4017-a362-6760fac66ee2-ovnkube-script-lib\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-cni-bin\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-run-netns\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.150231 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e257688-dca4-4017-a362-6760fac66ee2-env-overrides\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-node-log\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-run-openvswitch\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-registration-dir\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73781c1b-297f-4e84-a295-c720ff6c9b6c-iptables-alerter-script\") pod \"iptables-alerter-mjtg8\" (UID: \"73781c1b-297f-4e84-a295-c720ff6c9b6c\") " pod="openshift-network-operator/iptables-alerter-mjtg8"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1455f583-c317-4edc-98d3-9c8a7f14d37b-os-release\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-lib-modules\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149441    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2360ee4d-31bb-4de1-91ad-278352f88f22-host\") pod \"node-ca-9nzfl\" (UID: \"2360ee4d-31bb-4de1-91ad-278352f88f22\") " pod="openshift-image-registry/node-ca-9nzfl"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-run-netns\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-etc-selinux\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-log-socket\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1455f583-c317-4edc-98d3-9c8a7f14d37b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-etc-selinux\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/523ee06f-af02-40dd-8d8c-602f2f9a6452-socket-dir\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-var-lib-openvswitch\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-cnibin\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1455f583-c317-4edc-98d3-9c8a7f14d37b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.151042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149775    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.151855 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.149827    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs podName:2e0785e8-c533-44cb-bc6f-d11ed1bb60f0 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:03.649786769 +0000 UTC m=+3.025446063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs") pod "network-metrics-daemon-xkxwb" (UID: "2e0785e8-c533-44cb-bc6f-d11ed1bb60f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 18:40:03.151855 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.149992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1455f583-c317-4edc-98d3-9c8a7f14d37b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.151855 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.150034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73781c1b-297f-4e84-a295-c720ff6c9b6c-host-slash\") pod \"iptables-alerter-mjtg8\" (UID: \"73781c1b-297f-4e84-a295-c720ff6c9b6c\") " pod="openshift-network-operator/iptables-alerter-mjtg8"
Jul 24 18:40:03.151855 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.150063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpkgb\" (UniqueName: \"kubernetes.io/projected/3e257688-dca4-4017-a362-6760fac66ee2-kube-api-access-fpkgb\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.151855 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.150090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1455f583-c317-4edc-98d3-9c8a7f14d37b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.151855 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.150113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1455f583-c317-4edc-98d3-9c8a7f14d37b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.151855 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.150093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22cgz\" (UniqueName: \"kubernetes.io/projected/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-kube-api-access-22cgz\") pod \"network-metrics-daemon-xkxwb\" (UID: \"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0\") " pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:03.151855 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.150130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-run-systemd\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.151855 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.150212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73781c1b-297f-4e84-a295-c720ff6c9b6c-host-slash\") pod \"iptables-alerter-mjtg8\" (UID: \"73781c1b-297f-4e84-a295-c720ff6c9b6c\") " pod="openshift-network-operator/iptables-alerter-mjtg8"
Jul 24 18:40:03.151855 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.150235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e257688-dca4-4017-a362-6760fac66ee2-var-lib-openvswitch\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.151855 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.150557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e257688-dca4-4017-a362-6760fac66ee2-env-overrides\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.151855 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.150735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48692fa5-2ce7-4575-bacb-e0e6c5ca40a1-konnectivity-ca\") pod \"konnectivity-agent-6mlfh\" (UID: \"48692fa5-2ce7-4575-bacb-e0e6c5ca40a1\") " pod="kube-system/konnectivity-agent-6mlfh"
Jul 24 18:40:03.155304 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.155254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e257688-dca4-4017-a362-6760fac66ee2-ovn-node-metrics-cert\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.155304 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.155289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48692fa5-2ce7-4575-bacb-e0e6c5ca40a1-agent-certs\") pod \"konnectivity-agent-6mlfh\" (UID: \"48692fa5-2ce7-4575-bacb-e0e6c5ca40a1\") " pod="kube-system/konnectivity-agent-6mlfh"
Jul 24 18:40:03.158298 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.158273    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 18:40:03.158415 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.158314    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 18:40:03.158415 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.158328    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2sb9t for pod openshift-network-diagnostics/network-check-target-f6w5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 18:40:03.158415 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.158397    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t podName:d932da4a-af42-4d08-a40d-a1c2ce4e1123 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:03.658365725 +0000 UTC m=+3.034024998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2sb9t" (UniqueName: "kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t") pod "network-check-target-f6w5m" (UID: "d932da4a-af42-4d08-a40d-a1c2ce4e1123") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 18:40:03.159806 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.159744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74fqd\" (UniqueName: \"kubernetes.io/projected/2360ee4d-31bb-4de1-91ad-278352f88f22-kube-api-access-74fqd\") pod \"node-ca-9nzfl\" (UID: \"2360ee4d-31bb-4de1-91ad-278352f88f22\") " pod="openshift-image-registry/node-ca-9nzfl"
Jul 24 18:40:03.160118 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.160094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2tng\" (UniqueName: \"kubernetes.io/projected/73781c1b-297f-4e84-a295-c720ff6c9b6c-kube-api-access-q2tng\") pod \"iptables-alerter-mjtg8\" (UID: \"73781c1b-297f-4e84-a295-c720ff6c9b6c\") " pod="openshift-network-operator/iptables-alerter-mjtg8"
Jul 24 18:40:03.160292 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.160272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nknqv\" (UniqueName: \"kubernetes.io/projected/28bdcc77-a763-49e4-abb3-98de9bd1d399-kube-api-access-nknqv\") pod \"node-resolver-l9drv\" (UID: \"28bdcc77-a763-49e4-abb3-98de9bd1d399\") " pod="openshift-dns/node-resolver-l9drv"
Jul 24 18:40:03.162326 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.162187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnqbv\" (UniqueName: \"kubernetes.io/projected/1455f583-c317-4edc-98d3-9c8a7f14d37b-kube-api-access-vnqbv\") pod \"multus-additional-cni-plugins-9wq59\" (UID: \"1455f583-c317-4edc-98d3-9c8a7f14d37b\") " pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.162326 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.162208    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv5rx\" (UniqueName: \"kubernetes.io/projected/523ee06f-af02-40dd-8d8c-602f2f9a6452-kube-api-access-vv5rx\") pod \"aws-ebs-csi-driver-node-skw6b\" (UID: \"523ee06f-af02-40dd-8d8c-602f2f9a6452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.162326 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.162297    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpkgb\" (UniqueName: \"kubernetes.io/projected/3e257688-dca4-4017-a362-6760fac66ee2-kube-api-access-fpkgb\") pod \"ovnkube-node-hrxnr\" (UID: \"3e257688-dca4-4017-a362-6760fac66ee2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.163111 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.163090    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 18:40:03.164349 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.164317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22cgz\" (UniqueName: \"kubernetes.io/projected/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-kube-api-access-22cgz\") pod \"network-metrics-daemon-xkxwb\" (UID: \"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0\") " pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:03.195311 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.195223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal" event={"ID":"6550cfaced065c17322d923d252f38de","Type":"ContainerStarted","Data":"e74956b5b711e79f5ed58a24e2d7c21b00024e97ca17766e6ba87d5492bf2eef"}
Jul 24 18:40:03.196802 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.196755    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal" event={"ID":"05b93cd3ce9aed02e2f4eddde9bac7ee","Type":"ContainerStarted","Data":"67ca394a549cc8f766e7c233b5f100fd7d1b643156df6f3ab062a8d21649894a"}
Jul 24 18:40:03.250473 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-run\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.250587 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/403dce40-1510-40cb-a119-e5bba09ed289-tmp\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.250587 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-system-cni-dir\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.250587 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:03.250587 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/feb5428a-ec41-4135-9a40-5bff75da000b-cni-binary-copy\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.250789 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-run-k8s-cni-cncf-io\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.250789 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-etc-kubernetes\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.250789 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-modprobe-d\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.250789 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250666    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-sysconfig\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.250789 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-kubernetes\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.250789 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-hostroot\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.250789 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wprf\" (UniqueName: \"kubernetes.io/projected/feb5428a-ec41-4135-9a40-5bff75da000b-kube-api-access-7wprf\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.250789 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.250752    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 18:40:03.250789 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/403dce40-1510-40cb-a119-e5bba09ed289-etc-tuned\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.250814    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret podName:8bc4eada-1823-4bc2-add7-98ef884c80d4 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:03.750794529 +0000 UTC m=+3.126453810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret") pod "global-pull-secret-syncer-vn9rh" (UID: "8bc4eada-1823-4bc2-add7-98ef884c80d4") : object "kube-system"/"original-pull-secret" not registered
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-var-lib-kubelet\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bc4eada-1823-4bc2-add7-98ef884c80d4-dbus\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-sysctl-conf\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-host\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-os-release\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-run-multus-certs\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.250989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gn6g\" (UniqueName: \"kubernetes.io/projected/403dce40-1510-40cb-a119-e5bba09ed289-kube-api-access-9gn6g\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/feb5428a-ec41-4135-9a40-5bff75da000b-multus-daemon-config\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-sys\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-multus-cni-dir\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-var-lib-cni-bin\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-var-lib-cni-multus\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-modprobe-d\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-lib-modules\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-run-netns\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.251198 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-cnibin\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.252006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bc4eada-1823-4bc2-add7-98ef884c80d4-kubelet-config\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:03.252006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-var-lib-kubelet\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.252006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-multus-socket-dir-parent\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.252006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-multus-conf-dir\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.252006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-sysctl-d\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.252006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-systemd\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.252006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-systemd\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.252006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-run\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.252006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/feb5428a-ec41-4135-9a40-5bff75da000b-cni-binary-copy\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.252006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-run-k8s-cni-cncf-io\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.252006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-etc-kubernetes\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.252006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-sys\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.252006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.251849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-system-cni-dir\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.252734 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-var-lib-kubelet\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.252734 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bc4eada-1823-4bc2-add7-98ef884c80d4-dbus\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:03.252734 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-kubernetes\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.252734 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-sysconfig\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.252734 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-sysctl-conf\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.252734 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252296    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-hostroot\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.252734 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-host\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.252734 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-os-release\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.252734 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-run-multus-certs\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.252734 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-cnibin\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.252734 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252734    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-multus-cni-dir\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.253238 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-var-lib-cni-bin\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.253238 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-var-lib-cni-multus\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.253238 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-lib-modules\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.253238 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-host-run-netns\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.253238 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.252956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/feb5428a-ec41-4135-9a40-5bff75da000b-multus-daemon-config\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.253238 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.253004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-multus-socket-dir-parent\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.253238 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.253009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/feb5428a-ec41-4135-9a40-5bff75da000b-multus-conf-dir\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.253238 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.253053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bc4eada-1823-4bc2-add7-98ef884c80d4-kubelet-config\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:03.253238 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.253099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-var-lib-kubelet\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.253238 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.253099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/403dce40-1510-40cb-a119-e5bba09ed289-etc-sysctl-d\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.254847 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.254821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/403dce40-1510-40cb-a119-e5bba09ed289-tmp\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.255749 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.255716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/403dce40-1510-40cb-a119-e5bba09ed289-etc-tuned\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.261104 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.260980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wprf\" (UniqueName: \"kubernetes.io/projected/feb5428a-ec41-4135-9a40-5bff75da000b-kube-api-access-7wprf\") pod \"multus-pnpwv\" (UID: \"feb5428a-ec41-4135-9a40-5bff75da000b\") " pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.264924 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.264902    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gn6g\" (UniqueName: \"kubernetes.io/projected/403dce40-1510-40cb-a119-e5bba09ed289-kube-api-access-9gn6g\") pod \"tuned-mtfgj\" (UID: \"403dce40-1510-40cb-a119-e5bba09ed289\") " pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.327783 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.327753    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b"
Jul 24 18:40:03.337247 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.337226    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9nzfl"
Jul 24 18:40:03.341257 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:40:03.341217    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523ee06f_af02_40dd_8d8c_602f2f9a6452.slice/crio-b80dbce40e10971b42164bf36d673d69ab0e5a5fc21025568e10e3e638f222dc WatchSource:0}: Error finding container b80dbce40e10971b42164bf36d673d69ab0e5a5fc21025568e10e3e638f222dc: Status 404 returned error can't find the container with id b80dbce40e10971b42164bf36d673d69ab0e5a5fc21025568e10e3e638f222dc
Jul 24 18:40:03.347261 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.347238    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:03.349395 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:40:03.349353    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2360ee4d_31bb_4de1_91ad_278352f88f22.slice/crio-069a50c9bbbdfd2ea9009e59b87f9549e95a81785564ca533874e386c1cbd8b6 WatchSource:0}: Error finding container 069a50c9bbbdfd2ea9009e59b87f9549e95a81785564ca533874e386c1cbd8b6: Status 404 returned error can't find the container with id 069a50c9bbbdfd2ea9009e59b87f9549e95a81785564ca533874e386c1cbd8b6
Jul 24 18:40:03.350523 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.350505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9wq59"
Jul 24 18:40:03.358156 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.357985    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6mlfh"
Jul 24 18:40:03.361531 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:40:03.361491    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e257688_dca4_4017_a362_6760fac66ee2.slice/crio-498f0a19fecf17927a7b004da565595984ca126c09e4b4e141b7e15bc289944b WatchSource:0}: Error finding container 498f0a19fecf17927a7b004da565595984ca126c09e4b4e141b7e15bc289944b: Status 404 returned error can't find the container with id 498f0a19fecf17927a7b004da565595984ca126c09e4b4e141b7e15bc289944b
Jul 24 18:40:03.367472 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.367426    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mjtg8"
Jul 24 18:40:03.369700 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:40:03.369677    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1455f583_c317_4edc_98d3_9c8a7f14d37b.slice/crio-43419f3440130926f304a5229c51cb6e872c2c85aa53c9770fcc7b808b3c8b27 WatchSource:0}: Error finding container 43419f3440130926f304a5229c51cb6e872c2c85aa53c9770fcc7b808b3c8b27: Status 404 returned error can't find the container with id 43419f3440130926f304a5229c51cb6e872c2c85aa53c9770fcc7b808b3c8b27
Jul 24 18:40:03.373008 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:40:03.372980    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48692fa5_2ce7_4575_bacb_e0e6c5ca40a1.slice/crio-c0c9f70607ce592d7e6409be656939b0cbee35b54dcfea4b02f3b003e11f2abc WatchSource:0}: Error finding container c0c9f70607ce592d7e6409be656939b0cbee35b54dcfea4b02f3b003e11f2abc: Status 404 returned error can't find the container with id c0c9f70607ce592d7e6409be656939b0cbee35b54dcfea4b02f3b003e11f2abc
Jul 24 18:40:03.373847 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.373818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l9drv"
Jul 24 18:40:03.379880 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:40:03.379858    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73781c1b_297f_4e84_a295_c720ff6c9b6c.slice/crio-dacf12959ad28f04a7b32d7a694e98c2272ba5b5394c11e46113bea6f0f951df WatchSource:0}: Error finding container dacf12959ad28f04a7b32d7a694e98c2272ba5b5394c11e46113bea6f0f951df: Status 404 returned error can't find the container with id dacf12959ad28f04a7b32d7a694e98c2272ba5b5394c11e46113bea6f0f951df
Jul 24 18:40:03.382621 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.382547    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mtfgj"
Jul 24 18:40:03.384939 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:40:03.384914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28bdcc77_a763_49e4_abb3_98de9bd1d399.slice/crio-13f9fb520b4df531c5e337c1a610a488f864ec4854baa894d4cdb100ab475065 WatchSource:0}: Error finding container 13f9fb520b4df531c5e337c1a610a488f864ec4854baa894d4cdb100ab475065: Status 404 returned error can't find the container with id 13f9fb520b4df531c5e337c1a610a488f864ec4854baa894d4cdb100ab475065
Jul 24 18:40:03.388038 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.388013    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pnpwv"
Jul 24 18:40:03.396562 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:40:03.396540    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403dce40_1510_40cb_a119_e5bba09ed289.slice/crio-661ece1f2672ca85003ccb4358de403aef6d2e237f8c61ecdef93a050cb99808 WatchSource:0}: Error finding container 661ece1f2672ca85003ccb4358de403aef6d2e237f8c61ecdef93a050cb99808: Status 404 returned error can't find the container with id 661ece1f2672ca85003ccb4358de403aef6d2e237f8c61ecdef93a050cb99808
Jul 24 18:40:03.399204 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:40:03.399182    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb5428a_ec41_4135_9a40_5bff75da000b.slice/crio-00340dde8206b8668e2b4e58fb796c867f70da8070bd03a68194ecab3d18db6a WatchSource:0}: Error finding container 00340dde8206b8668e2b4e58fb796c867f70da8070bd03a68194ecab3d18db6a: Status 404 returned error can't find the container with id 00340dde8206b8668e2b4e58fb796c867f70da8070bd03a68194ecab3d18db6a
Jul 24 18:40:03.653904 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.653820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs\") pod \"network-metrics-daemon-xkxwb\" (UID: \"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0\") " pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:03.654036 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.653988    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 18:40:03.654101 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.654068    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs podName:2e0785e8-c533-44cb-bc6f-d11ed1bb60f0 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:04.654048463 +0000 UTC m=+4.029707761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs") pod "network-metrics-daemon-xkxwb" (UID: "2e0785e8-c533-44cb-bc6f-d11ed1bb60f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 18:40:03.755178 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.755142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:03.755178 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:03.755186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb9t\" (UniqueName: \"kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t\") pod \"network-check-target-f6w5m\" (UID: \"d932da4a-af42-4d08-a40d-a1c2ce4e1123\") " pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:03.755375 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.755303    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 18:40:03.755375 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.755365    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret podName:8bc4eada-1823-4bc2-add7-98ef884c80d4 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:04.755349946 +0000 UTC m=+4.131009219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret") pod "global-pull-secret-syncer-vn9rh" (UID: "8bc4eada-1823-4bc2-add7-98ef884c80d4") : object "kube-system"/"original-pull-secret" not registered
Jul 24 18:40:03.755501 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.755309    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 18:40:03.755501 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.755412    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 18:40:03.755501 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.755425    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2sb9t for pod openshift-network-diagnostics/network-check-target-f6w5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 18:40:03.755501 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:03.755473    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t podName:d932da4a-af42-4d08-a40d-a1c2ce4e1123 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:04.755459727 +0000 UTC m=+4.131119018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2sb9t" (UniqueName: "kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t") pod "network-check-target-f6w5m" (UID: "d932da4a-af42-4d08-a40d-a1c2ce4e1123") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 18:40:04.084767 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.084666    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 18:35:03 +0000 UTC" deadline="2028-05-02 06:18:12.122673802 +0000 UTC"
Jul 24 18:40:04.084767 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.084718    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15539h38m8.037959838s"
Jul 24 18:40:04.120109 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.119778    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 18:40:04.191417 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.191343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:04.191595 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:04.191497    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:04.200284 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.200231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pnpwv" event={"ID":"feb5428a-ec41-4135-9a40-5bff75da000b","Type":"ContainerStarted","Data":"00340dde8206b8668e2b4e58fb796c867f70da8070bd03a68194ecab3d18db6a"}
Jul 24 18:40:04.201853 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.201802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mtfgj" event={"ID":"403dce40-1510-40cb-a119-e5bba09ed289","Type":"ContainerStarted","Data":"661ece1f2672ca85003ccb4358de403aef6d2e237f8c61ecdef93a050cb99808"}
Jul 24 18:40:04.203230 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.203173    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l9drv" event={"ID":"28bdcc77-a763-49e4-abb3-98de9bd1d399","Type":"ContainerStarted","Data":"13f9fb520b4df531c5e337c1a610a488f864ec4854baa894d4cdb100ab475065"}
Jul 24 18:40:04.204515 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.204477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6mlfh" event={"ID":"48692fa5-2ce7-4575-bacb-e0e6c5ca40a1","Type":"ContainerStarted","Data":"c0c9f70607ce592d7e6409be656939b0cbee35b54dcfea4b02f3b003e11f2abc"}
Jul 24 18:40:04.206196 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.206134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9wq59" event={"ID":"1455f583-c317-4edc-98d3-9c8a7f14d37b","Type":"ContainerStarted","Data":"43419f3440130926f304a5229c51cb6e872c2c85aa53c9770fcc7b808b3c8b27"}
Jul 24 18:40:04.207833 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.207730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr" event={"ID":"3e257688-dca4-4017-a362-6760fac66ee2","Type":"ContainerStarted","Data":"498f0a19fecf17927a7b004da565595984ca126c09e4b4e141b7e15bc289944b"}
Jul 24 18:40:04.209404 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.209365    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mjtg8" event={"ID":"73781c1b-297f-4e84-a295-c720ff6c9b6c","Type":"ContainerStarted","Data":"dacf12959ad28f04a7b32d7a694e98c2272ba5b5394c11e46113bea6f0f951df"}
Jul 24 18:40:04.210994 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.210967    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9nzfl" event={"ID":"2360ee4d-31bb-4de1-91ad-278352f88f22","Type":"ContainerStarted","Data":"069a50c9bbbdfd2ea9009e59b87f9549e95a81785564ca533874e386c1cbd8b6"}
Jul 24 18:40:04.212283 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.212264    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b" event={"ID":"523ee06f-af02-40dd-8d8c-602f2f9a6452","Type":"ContainerStarted","Data":"b80dbce40e10971b42164bf36d673d69ab0e5a5fc21025568e10e3e638f222dc"}
Jul 24 18:40:04.663165 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.663125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs\") pod \"network-metrics-daemon-xkxwb\" (UID: \"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0\") " pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:04.663767 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:04.663348    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 18:40:04.663767 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:04.663430    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs podName:2e0785e8-c533-44cb-bc6f-d11ed1bb60f0 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:06.663410563 +0000 UTC m=+6.039069843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs") pod "network-metrics-daemon-xkxwb" (UID: "2e0785e8-c533-44cb-bc6f-d11ed1bb60f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 18:40:04.764832 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.764542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:04.764832 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:04.764609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb9t\" (UniqueName: \"kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t\") pod \"network-check-target-f6w5m\" (UID: \"d932da4a-af42-4d08-a40d-a1c2ce4e1123\") " pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:04.764832 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:04.764662    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 18:40:04.764832 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:04.764722    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret podName:8bc4eada-1823-4bc2-add7-98ef884c80d4 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:06.764705235 +0000 UTC m=+6.140364525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret") pod "global-pull-secret-syncer-vn9rh" (UID: "8bc4eada-1823-4bc2-add7-98ef884c80d4") : object "kube-system"/"original-pull-secret" not registered
Jul 24 18:40:04.764832 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:04.764745    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 18:40:04.764832 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:04.764759    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 18:40:04.764832 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:04.764771    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2sb9t for pod openshift-network-diagnostics/network-check-target-f6w5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 18:40:04.764832 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:04.764806    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t podName:d932da4a-af42-4d08-a40d-a1c2ce4e1123 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:06.764794992 +0000 UTC m=+6.140454277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2sb9t" (UniqueName: "kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t") pod "network-check-target-f6w5m" (UID: "d932da4a-af42-4d08-a40d-a1c2ce4e1123") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 18:40:05.085941 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:05.085877    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 18:35:03 +0000 UTC" deadline="2028-01-17 12:20:47.727809385 +0000 UTC"
Jul 24 18:40:05.085941 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:05.085913    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13001h40m42.641900997s"
Jul 24 18:40:05.196275 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:05.196072    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:05.196443 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:05.196373    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:05.196519 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:05.196456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:05.196588 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:05.196565    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:05.223629 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:05.223597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal" event={"ID":"6550cfaced065c17322d923d252f38de","Type":"ContainerStarted","Data":"6617bb2ee273664889aafec60c4a634f3f2e57f0bad8efd8a9ab1a4034934ae0"}
Jul 24 18:40:05.238883 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:05.237764    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-111.ec2.internal" podStartSLOduration=3.237745919 podStartE2EDuration="3.237745919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:40:05.23758434 +0000 UTC m=+4.613243636" watchObservedRunningTime="2026-07-24 18:40:05.237745919 +0000 UTC m=+4.613405216"
Jul 24 18:40:06.190675 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:06.190640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:06.191117 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:06.190765    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:06.232592 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:06.232556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal" event={"ID":"05b93cd3ce9aed02e2f4eddde9bac7ee","Type":"ContainerStarted","Data":"f54ec18ee195e591413c0f1b5422e19f33a7b1b4ee5f4e7065873005c326e9d7"}
Jul 24 18:40:06.682702 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:06.682658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs\") pod \"network-metrics-daemon-xkxwb\" (UID: \"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0\") " pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:06.682883 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:06.682826    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 18:40:06.682946 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:06.682887    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs podName:2e0785e8-c533-44cb-bc6f-d11ed1bb60f0 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:10.682868436 +0000 UTC m=+10.058527716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs") pod "network-metrics-daemon-xkxwb" (UID: "2e0785e8-c533-44cb-bc6f-d11ed1bb60f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 18:40:06.783039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:06.783003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb9t\" (UniqueName: \"kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t\") pod \"network-check-target-f6w5m\" (UID: \"d932da4a-af42-4d08-a40d-a1c2ce4e1123\") " pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:06.783219 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:06.783087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:06.783219 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:06.783170    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 18:40:06.783219 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:06.783190    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 18:40:06.783219 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:06.783204    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2sb9t for pod openshift-network-diagnostics/network-check-target-f6w5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 18:40:06.783436 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:06.783260    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t podName:d932da4a-af42-4d08-a40d-a1c2ce4e1123 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:10.783240745 +0000 UTC m=+10.158900033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2sb9t" (UniqueName: "kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t") pod "network-check-target-f6w5m" (UID: "d932da4a-af42-4d08-a40d-a1c2ce4e1123") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 18:40:06.784362 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:06.783175    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 18:40:06.784362 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:06.783632    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret podName:8bc4eada-1823-4bc2-add7-98ef884c80d4 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:10.783614357 +0000 UTC m=+10.159273635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret") pod "global-pull-secret-syncer-vn9rh" (UID: "8bc4eada-1823-4bc2-add7-98ef884c80d4") : object "kube-system"/"original-pull-secret" not registered
Jul 24 18:40:07.191447 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:07.191413    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:07.191883 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:07.191455    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:07.191883 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:07.191543    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:07.192009 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:07.191957    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:08.191326 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:08.191298    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:08.191517 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:08.191452    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:08.239480 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:08.239414    2580 generic.go:356] "Generic (PLEG): container finished" podID="05b93cd3ce9aed02e2f4eddde9bac7ee" containerID="f54ec18ee195e591413c0f1b5422e19f33a7b1b4ee5f4e7065873005c326e9d7" exitCode=0
Jul 24 18:40:08.239480 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:08.239467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal" event={"ID":"05b93cd3ce9aed02e2f4eddde9bac7ee","Type":"ContainerDied","Data":"f54ec18ee195e591413c0f1b5422e19f33a7b1b4ee5f4e7065873005c326e9d7"}
Jul 24 18:40:09.191122 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:09.191089    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:09.191306 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:09.191089    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:09.191375 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:09.191332    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:09.191375 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:09.191230    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:10.191303 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:10.191272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:10.191780 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:10.191410    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:10.721496 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:10.721452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs\") pod \"network-metrics-daemon-xkxwb\" (UID: \"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0\") " pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:10.721685 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:10.721615    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 18:40:10.721685 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:10.721682    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs podName:2e0785e8-c533-44cb-bc6f-d11ed1bb60f0 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:18.721664278 +0000 UTC m=+18.097323555 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs") pod "network-metrics-daemon-xkxwb" (UID: "2e0785e8-c533-44cb-bc6f-d11ed1bb60f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 18:40:10.822008 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:10.821965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:10.822194 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:10.822040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb9t\" (UniqueName: \"kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t\") pod \"network-check-target-f6w5m\" (UID: \"d932da4a-af42-4d08-a40d-a1c2ce4e1123\") " pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:10.822259 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:10.822192    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 18:40:10.822259 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:10.822213    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 18:40:10.822259 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:10.822226    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2sb9t for pod openshift-network-diagnostics/network-check-target-f6w5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 18:40:10.822427 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:10.822284    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t podName:d932da4a-af42-4d08-a40d-a1c2ce4e1123 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:18.822266621 +0000 UTC m=+18.197925923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2sb9t" (UniqueName: "kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t") pod "network-check-target-f6w5m" (UID: "d932da4a-af42-4d08-a40d-a1c2ce4e1123") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 18:40:10.822503 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:10.822475    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 18:40:10.822618 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:10.822547    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret podName:8bc4eada-1823-4bc2-add7-98ef884c80d4 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:18.822527201 +0000 UTC m=+18.198186497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret") pod "global-pull-secret-syncer-vn9rh" (UID: "8bc4eada-1823-4bc2-add7-98ef884c80d4") : object "kube-system"/"original-pull-secret" not registered
Jul 24 18:40:11.195450 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:11.194600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:11.195450 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:11.194783    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:11.195450 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:11.195220    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:11.195450 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:11.195350    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:12.191356 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:12.191148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:12.191356 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:12.191270    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:13.191094 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:13.191062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:13.191541 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:13.191184    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:13.191620 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:13.191570    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:13.191712 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:13.191681    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:14.190717 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:14.190638    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:14.190860 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:14.190762    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:15.191328 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:15.191297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:15.191838 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:15.191456    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:15.191838 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:15.191506    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:15.191838 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:15.191616    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:16.190824 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:16.190786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:16.190992 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:16.190904    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:17.190879 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:17.190845    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:17.191360 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:17.190845    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:17.191360 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:17.190993    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:17.191360 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:17.191030    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:18.191523 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:18.191488    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:18.192027 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:18.191608    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:18.782840 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:18.782809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs\") pod \"network-metrics-daemon-xkxwb\" (UID: \"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0\") " pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:18.783020 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:18.782932    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 18:40:18.783020 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:18.782987    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs podName:2e0785e8-c533-44cb-bc6f-d11ed1bb60f0 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:34.782968877 +0000 UTC m=+34.158628156 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs") pod "network-metrics-daemon-xkxwb" (UID: "2e0785e8-c533-44cb-bc6f-d11ed1bb60f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 18:40:18.883569 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:18.883539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:18.883740 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:18.883592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb9t\" (UniqueName: \"kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t\") pod \"network-check-target-f6w5m\" (UID: \"d932da4a-af42-4d08-a40d-a1c2ce4e1123\") " pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:18.883740 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:18.883724    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 18:40:18.883844 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:18.883811    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret podName:8bc4eada-1823-4bc2-add7-98ef884c80d4 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:34.883791449 +0000 UTC m=+34.259450807 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret") pod "global-pull-secret-syncer-vn9rh" (UID: "8bc4eada-1823-4bc2-add7-98ef884c80d4") : object "kube-system"/"original-pull-secret" not registered
Jul 24 18:40:18.883844 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:18.883728    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 18:40:18.883931 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:18.883867    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 18:40:18.883931 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:18.883883    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2sb9t for pod openshift-network-diagnostics/network-check-target-f6w5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 18:40:18.884027 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:18.883946    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t podName:d932da4a-af42-4d08-a40d-a1c2ce4e1123 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:34.883933442 +0000 UTC m=+34.259592735 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2sb9t" (UniqueName: "kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t") pod "network-check-target-f6w5m" (UID: "d932da4a-af42-4d08-a40d-a1c2ce4e1123") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 18:40:19.191459 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:19.191374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:19.191622 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:19.191517    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:19.191622 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:19.191583    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:19.192011 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:19.191688    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:20.190568 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:20.190531    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:20.190737 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:20.190644    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:21.192047 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:21.192017    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:21.192505 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:21.192134    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:21.192505 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:21.192148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:21.192505 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:21.192250    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:22.190758 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.190727    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:22.190870 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:22.190824    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:22.267510 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.267476    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pnpwv" event={"ID":"feb5428a-ec41-4135-9a40-5bff75da000b","Type":"ContainerStarted","Data":"b99123502d2b9c267794d566b302506354906c43dae4b54e03a802b5e7c25ea7"}
Jul 24 18:40:22.269607 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.269577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mtfgj" event={"ID":"403dce40-1510-40cb-a119-e5bba09ed289","Type":"ContainerStarted","Data":"b93f1dc9ce0161cfbceffa28136c1fe92ec37719f3173b2ba2bb2d5cea4317ed"}
Jul 24 18:40:22.270989 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.270963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l9drv" event={"ID":"28bdcc77-a763-49e4-abb3-98de9bd1d399","Type":"ContainerStarted","Data":"f134e56d7d569a425b10d74d647f7c8488b56044e4e1df4e354a5475b971c947"}
Jul 24 18:40:22.272371 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.272344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6mlfh" event={"ID":"48692fa5-2ce7-4575-bacb-e0e6c5ca40a1","Type":"ContainerStarted","Data":"39409a00507b48299b54d3d822ff62f1e08f61fbc5dcebb5b4a895ff8a75e087"}
Jul 24 18:40:22.273693 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.273672    2580 generic.go:356] "Generic (PLEG): container finished" podID="1455f583-c317-4edc-98d3-9c8a7f14d37b" containerID="4f6082f6c5a5764e5f8bac4c5f28a4430ebca21142de9a329025b3ee388b93d7" exitCode=0
Jul 24 18:40:22.273787 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.273734    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9wq59" event={"ID":"1455f583-c317-4edc-98d3-9c8a7f14d37b","Type":"ContainerDied","Data":"4f6082f6c5a5764e5f8bac4c5f28a4430ebca21142de9a329025b3ee388b93d7"}
Jul 24 18:40:22.274954 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.274933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9nzfl" event={"ID":"2360ee4d-31bb-4de1-91ad-278352f88f22","Type":"ContainerStarted","Data":"812450dd208b302ea60a35bbd7cda1409bcefc3455664ecaed672e00824a4d51"}
Jul 24 18:40:22.276214 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.276196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b" event={"ID":"523ee06f-af02-40dd-8d8c-602f2f9a6452","Type":"ContainerStarted","Data":"81799701ccaa127c7bbaf6fffa9c0a94298d93d661412c6c3932efb840b91200"}
Jul 24 18:40:22.277702 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.277683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal" event={"ID":"05b93cd3ce9aed02e2f4eddde9bac7ee","Type":"ContainerStarted","Data":"c8012d21cf0a489a19a027733eeb2117804c6596d07bba42e52052735b5e070d"}
Jul 24 18:40:22.285591 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.285556    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pnpwv" podStartSLOduration=3.029536955 podStartE2EDuration="21.285545138s" totalImagesPullingTime="18.256008183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:40:22.28543991 +0000 UTC m=+21.661099206" watchObservedRunningTime="2026-07-24 18:40:22.285545138 +0000 UTC m=+21.661204435"
Jul 24 18:40:22.298466 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.298432    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9nzfl" podStartSLOduration=3.202792666 podStartE2EDuration="21.298421007s" totalImagesPullingTime="18.095628341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:40:22.298400351 +0000 UTC m=+21.674059639" watchObservedRunningTime="2026-07-24 18:40:22.298421007 +0000 UTC m=+21.674080303"
Jul 24 18:40:22.310930 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.310894    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6mlfh" podStartSLOduration=3.031669326 podStartE2EDuration="21.310878984s" totalImagesPullingTime="18.279209658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:40:22.310756389 +0000 UTC m=+21.686415684" watchObservedRunningTime="2026-07-24 18:40:22.310878984 +0000 UTC m=+21.686538280"
Jul 24 18:40:22.345950 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.345917    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l9drv" podStartSLOduration=3.286001906 podStartE2EDuration="21.345906882s" totalImagesPullingTime="18.059904976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:40:22.345659863 +0000 UTC m=+21.721319159" watchObservedRunningTime="2026-07-24 18:40:22.345906882 +0000 UTC m=+21.721566177"
Jul 24 18:40:22.359755 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.359723    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mtfgj" podStartSLOduration=3.102748026 podStartE2EDuration="21.359713238s" totalImagesPullingTime="18.256965212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:40:22.359622256 +0000 UTC m=+21.735281551" watchObservedRunningTime="2026-07-24 18:40:22.359713238 +0000 UTC m=+21.735372533"
Jul 24 18:40:22.371914 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:22.371885    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-111.ec2.internal" podStartSLOduration=20.371873569 podStartE2EDuration="20.371873569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:40:22.371643325 +0000 UTC m=+21.747302621" watchObservedRunningTime="2026-07-24 18:40:22.371873569 +0000 UTC m=+21.747532864"
Jul 24 18:40:23.191126 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:23.191097    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:23.191277 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:23.191238    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:23.191357 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:23.191336    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:23.191489 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:23.191466    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:23.230504 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:23.230478    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 18:40:23.284812 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:23.284775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr" event={"ID":"3e257688-dca4-4017-a362-6760fac66ee2","Type":"ContainerStarted","Data":"0f461c298466fd2443fecdfec65995c09929e4a12d2c65a85e947ab50b8393d0"}
Jul 24 18:40:23.285293 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:23.284862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr" event={"ID":"3e257688-dca4-4017-a362-6760fac66ee2","Type":"ContainerStarted","Data":"cf27be99d6e73429641cc2c6699d0b1d959736f67eaef18daf36f01804935791"}
Jul 24 18:40:23.285293 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:23.284901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr" event={"ID":"3e257688-dca4-4017-a362-6760fac66ee2","Type":"ContainerStarted","Data":"38d1659ecd0efe0283ded740c468b81d2b425d84c1afbabeeb9e9a53549c3745"}
Jul 24 18:40:23.285293 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:23.284942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr" event={"ID":"3e257688-dca4-4017-a362-6760fac66ee2","Type":"ContainerStarted","Data":"8a844c7ba36153dbb7cad1ddcd40a3abb9d30c0a1fad2fc90d992a90a52e7486"}
Jul 24 18:40:23.285293 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:23.284988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr" event={"ID":"3e257688-dca4-4017-a362-6760fac66ee2","Type":"ContainerStarted","Data":"d628a1b6fa5e9ee644a1e1ed7372066aa85f71f57dac4f9857ea71d13c0659e8"}
Jul 24 18:40:23.285293 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:23.285034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr" event={"ID":"3e257688-dca4-4017-a362-6760fac66ee2","Type":"ContainerStarted","Data":"29c4a8317b44187328cc877884f50dcd944c4008154517bcb8723ff56d73f8f0"}
Jul 24 18:40:23.286074 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:23.286047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mjtg8" event={"ID":"73781c1b-297f-4e84-a295-c720ff6c9b6c","Type":"ContainerStarted","Data":"8d05cc54b05271c39f54aa9a5a900b1abe9c08c9f309308e2f2af8fec0a64fa5"}
Jul 24 18:40:23.287563 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:23.287538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b" event={"ID":"523ee06f-af02-40dd-8d8c-602f2f9a6452","Type":"ContainerStarted","Data":"3fe2d8394244b46c2c5d7a748e6fd8b36452da16115c68dad4ea72bb42457a2f"}
Jul 24 18:40:23.318101 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:23.318056    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mjtg8" podStartSLOduration=4.058424331 podStartE2EDuration="22.318040698s" totalImagesPullingTime="18.259616367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:40:23.316919488 +0000 UTC m=+22.692578794" watchObservedRunningTime="2026-07-24 18:40:23.318040698 +0000 UTC m=+22.693699996"
Jul 24 18:40:24.102569 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:24.102492    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T18:40:23.230500546Z","UUID":"0c67db1a-d340-4cea-b95d-e03f340a4ef3","Handler":null,"Name":"","Endpoint":""}
Jul 24 18:40:24.104083 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:24.104062    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 18:40:24.104176 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:24.104106    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 18:40:24.191480 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:24.191457    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:24.191570 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:24.191552    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:24.291498 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:24.291455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b" event={"ID":"523ee06f-af02-40dd-8d8c-602f2f9a6452","Type":"ContainerStarted","Data":"b649458a607c5c5b36d329f47e741c3ff8985d2c4c41657a1fca8e2542d7ae91"}
Jul 24 18:40:24.307825 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:24.307772    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skw6b" podStartSLOduration=3.074739377 podStartE2EDuration="23.307757985s" totalImagesPullingTime="20.233018608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:40:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:40:24.307209663 +0000 UTC m=+23.682868962" watchObservedRunningTime="2026-07-24 18:40:24.307757985 +0000 UTC m=+23.683417295"
Jul 24 18:40:25.191611 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:25.191412    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:25.191713 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:25.191441    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:25.191776 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:25.191727    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:25.191833 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:25.191816    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:25.298917 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:25.298884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr" event={"ID":"3e257688-dca4-4017-a362-6760fac66ee2","Type":"ContainerStarted","Data":"bf9feb75fa82feba7a394ea101464d991721fda40b80b27a8eec55dbeae11437"}
Jul 24 18:40:25.693144 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:25.693113    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6mlfh"
Jul 24 18:40:25.693716 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:25.693696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6mlfh"
Jul 24 18:40:26.191057 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:26.190976    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:26.191217 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:26.191075    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:26.301443 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:26.301413    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6mlfh"
Jul 24 18:40:26.302099 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:26.302081    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6mlfh"
Jul 24 18:40:27.191024 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:27.190993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:27.191184 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:27.191115    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:27.191184 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:27.191154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:27.191314 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:27.191242    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:28.191141 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:28.191107    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:28.191667 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:28.191238    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:29.191587 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:29.191559    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:29.191897 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:29.191596    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:29.191897 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:29.191686    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:29.191897 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:29.191791    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:29.308129 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:29.308068    2580 generic.go:356] "Generic (PLEG): container finished" podID="1455f583-c317-4edc-98d3-9c8a7f14d37b" containerID="4888b5e5a067aa78c6020f12c0d1b485d20263b640d9cb52985359b00c159fd7" exitCode=0
Jul 24 18:40:29.308248 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:29.308141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9wq59" event={"ID":"1455f583-c317-4edc-98d3-9c8a7f14d37b","Type":"ContainerDied","Data":"4888b5e5a067aa78c6020f12c0d1b485d20263b640d9cb52985359b00c159fd7"}
Jul 24 18:40:29.311834 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:29.311808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr" event={"ID":"3e257688-dca4-4017-a362-6760fac66ee2","Type":"ContainerStarted","Data":"a47275fd314d20fedb3986a0f804ea7933f6da8c451aefc02adccfdc8ab529c9"}
Jul 24 18:40:29.312006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:29.311990    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:29.312075 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:29.312067    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:29.326690 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:29.326671    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:29.351919 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:29.351884    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr" podStartSLOduration=9.469883342 podStartE2EDuration="28.351871481s" totalImagesPullingTime="18.881988139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:40:29.351053357 +0000 UTC m=+28.726712654" watchObservedRunningTime="2026-07-24 18:40:29.351871481 +0000 UTC m=+28.727530792"
Jul 24 18:40:30.190689 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:30.190655    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:30.190832 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:30.190794    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:30.313885 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:30.313852    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:30.328418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:30.328393    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:40:30.755131 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:30.755057    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f6w5m"]
Jul 24 18:40:30.755262 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:30.755217    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:30.755398 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:30.755353    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:30.758228 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:30.758204    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xkxwb"]
Jul 24 18:40:30.758335 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:30.758297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:30.758407 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:30.758391    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:30.769639 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:30.769618    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vn9rh"]
Jul 24 18:40:30.769709 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:30.769689    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:30.769761 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:30.769746    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:31.318734 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:31.318579    2580 generic.go:356] "Generic (PLEG): container finished" podID="1455f583-c317-4edc-98d3-9c8a7f14d37b" containerID="a704959b1a1d0a4d89d250578e23b0a6e7ca375c2159f0d55c17777a73638fa0" exitCode=0
Jul 24 18:40:31.319084 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:31.318655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9wq59" event={"ID":"1455f583-c317-4edc-98d3-9c8a7f14d37b","Type":"ContainerDied","Data":"a704959b1a1d0a4d89d250578e23b0a6e7ca375c2159f0d55c17777a73638fa0"}
Jul 24 18:40:32.191001 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:32.190931    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:32.191143 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:32.190931    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:32.191143 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:32.191025    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:32.191223 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:32.191134    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:32.191223 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:32.190931    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:32.191290 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:32.191230    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:33.325220 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:33.325184    2580 generic.go:356] "Generic (PLEG): container finished" podID="1455f583-c317-4edc-98d3-9c8a7f14d37b" containerID="138437a95cec660fcae747fb30acb230e1bf3616c69dafcacc8620f3f07c1817" exitCode=0
Jul 24 18:40:33.325705 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:33.325249    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9wq59" event={"ID":"1455f583-c317-4edc-98d3-9c8a7f14d37b","Type":"ContainerDied","Data":"138437a95cec660fcae747fb30acb230e1bf3616c69dafcacc8620f3f07c1817"}
Jul 24 18:40:34.191025 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:34.190994    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:34.191282 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:34.191032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:34.191282 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:34.191096    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:34.191282 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:34.191108    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6w5m" podUID="d932da4a-af42-4d08-a40d-a1c2ce4e1123"
Jul 24 18:40:34.191282 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:34.191175    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkxwb" podUID="2e0785e8-c533-44cb-bc6f-d11ed1bb60f0"
Jul 24 18:40:34.191282 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:34.191234    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vn9rh" podUID="8bc4eada-1823-4bc2-add7-98ef884c80d4"
Jul 24 18:40:34.808096 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:34.808066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs\") pod \"network-metrics-daemon-xkxwb\" (UID: \"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0\") " pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:34.808588 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:34.808173    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 18:40:34.808588 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:34.808221    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs podName:2e0785e8-c533-44cb-bc6f-d11ed1bb60f0 nodeName:}" failed. No retries permitted until 2026-07-24 18:41:06.808208452 +0000 UTC m=+66.183867726 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs") pod "network-metrics-daemon-xkxwb" (UID: "2e0785e8-c533-44cb-bc6f-d11ed1bb60f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 18:40:34.909206 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:34.909176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:34.909360 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:34.909221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb9t\" (UniqueName: \"kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t\") pod \"network-check-target-f6w5m\" (UID: \"d932da4a-af42-4d08-a40d-a1c2ce4e1123\") " pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:34.909360 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:34.909333    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 18:40:34.909360 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:34.909351    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 18:40:34.909495 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:34.909371    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 18:40:34.909495 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:34.909398    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2sb9t for pod openshift-network-diagnostics/network-check-target-f6w5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 18:40:34.909495 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:34.909416    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret podName:8bc4eada-1823-4bc2-add7-98ef884c80d4 nodeName:}" failed. No retries permitted until 2026-07-24 18:41:06.909396901 +0000 UTC m=+66.285056190 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret") pod "global-pull-secret-syncer-vn9rh" (UID: "8bc4eada-1823-4bc2-add7-98ef884c80d4") : object "kube-system"/"original-pull-secret" not registered
Jul 24 18:40:34.909495 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:34.909444    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t podName:d932da4a-af42-4d08-a40d-a1c2ce4e1123 nodeName:}" failed. No retries permitted until 2026-07-24 18:41:06.909429543 +0000 UTC m=+66.285088833 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2sb9t" (UniqueName: "kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t") pod "network-check-target-f6w5m" (UID: "d932da4a-af42-4d08-a40d-a1c2ce4e1123") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 18:40:35.396633 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.396595    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-111.ec2.internal" event="NodeReady"
Jul 24 18:40:35.396805 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.396733    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 18:40:35.430536 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.430472    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f59749bb5-kfnt9"]
Jul 24 18:40:35.457203 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.457179    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f59749bb5-kfnt9"]
Jul 24 18:40:35.457203 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.457209    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pfdnx"]
Jul 24 18:40:35.457360 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.457299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.459292 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.459265    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 18:40:35.459292 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.459287    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tlbzw\""
Jul 24 18:40:35.459481 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.459312    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 18:40:35.459481 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.459330    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 18:40:35.467672 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.467630    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 18:40:35.483208 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.483185    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b2cwr"]
Jul 24 18:40:35.483317 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.483271    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:35.485209 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.485190    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nwwl2\""
Jul 24 18:40:35.485323 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.485231    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 18:40:35.485323 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.485303    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 18:40:35.507299 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.507276    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b2cwr"]
Jul 24 18:40:35.507405 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.507314    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pfdnx"]
Jul 24 18:40:35.507461 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.507423    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b2cwr"
Jul 24 18:40:35.509361 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.509339    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 18:40:35.509461 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.509360    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 18:40:35.509461 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.509376    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 18:40:35.509461 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.509404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zw859\""
Jul 24 18:40:35.614393 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.614349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44c73487-26c9-4984-a178-1b20cf791453-config-volume\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:35.614393 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.614393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99213b47-96d3-4c66-8007-33c8b2fe15d7-cert\") pod \"ingress-canary-b2cwr\" (UID: \"99213b47-96d3-4c66-8007-33c8b2fe15d7\") " pod="openshift-ingress-canary/ingress-canary-b2cwr"
Jul 24 18:40:35.614573 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.614440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-bound-sa-token\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.614573 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.614482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:35.614573 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.614506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.614573 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.614532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-certificates\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.614573 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.614551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-installation-pull-secrets\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.614765 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.614573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-trusted-ca\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.614765 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.614616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-image-registry-private-configuration\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.614765 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.614652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-ca-trust-extracted\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.614765 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.614669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqmqk\" (UniqueName: \"kubernetes.io/projected/99213b47-96d3-4c66-8007-33c8b2fe15d7-kube-api-access-rqmqk\") pod \"ingress-canary-b2cwr\" (UID: \"99213b47-96d3-4c66-8007-33c8b2fe15d7\") " pod="openshift-ingress-canary/ingress-canary-b2cwr"
Jul 24 18:40:35.614765 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.614688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t277\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-kube-api-access-2t277\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.614765 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.614709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzrmm\" (UniqueName: \"kubernetes.io/projected/44c73487-26c9-4984-a178-1b20cf791453-kube-api-access-pzrmm\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:35.614765 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.614737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c73487-26c9-4984-a178-1b20cf791453-tmp-dir\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:35.715342 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c73487-26c9-4984-a178-1b20cf791453-tmp-dir\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:35.715342 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44c73487-26c9-4984-a178-1b20cf791453-config-volume\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:35.715559 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99213b47-96d3-4c66-8007-33c8b2fe15d7-cert\") pod \"ingress-canary-b2cwr\" (UID: \"99213b47-96d3-4c66-8007-33c8b2fe15d7\") " pod="openshift-ingress-canary/ingress-canary-b2cwr"
Jul 24 18:40:35.715559 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-bound-sa-token\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.715655 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:35.715706 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c73487-26c9-4984-a178-1b20cf791453-tmp-dir\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:35.715706 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.715810 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-certificates\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.715810 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-installation-pull-secrets\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.715810 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:35.715761    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 18:40:35.715810 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:35.715775    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f59749bb5-kfnt9: secret "image-registry-tls" not found
Jul 24 18:40:35.715810 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-trusted-ca\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.715810 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-image-registry-private-configuration\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.716107 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:35.715837    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls podName:c1cd83d8-1c8d-468c-9ea2-8661cf2eecac nodeName:}" failed. No retries permitted until 2026-07-24 18:40:36.215818553 +0000 UTC m=+35.591477846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls") pod "image-registry-5f59749bb5-kfnt9" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac") : secret "image-registry-tls" not found
Jul 24 18:40:35.716107 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-ca-trust-extracted\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.716107 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqmqk\" (UniqueName: \"kubernetes.io/projected/99213b47-96d3-4c66-8007-33c8b2fe15d7-kube-api-access-rqmqk\") pod \"ingress-canary-b2cwr\" (UID: \"99213b47-96d3-4c66-8007-33c8b2fe15d7\") " pod="openshift-ingress-canary/ingress-canary-b2cwr"
Jul 24 18:40:35.716107 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t277\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-kube-api-access-2t277\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.716107 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.715949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzrmm\" (UniqueName: \"kubernetes.io/projected/44c73487-26c9-4984-a178-1b20cf791453-kube-api-access-pzrmm\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:35.716363 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.716162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44c73487-26c9-4984-a178-1b20cf791453-config-volume\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:35.716363 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:35.716206    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 18:40:35.716363 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:35.716266    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls podName:44c73487-26c9-4984-a178-1b20cf791453 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:36.216250098 +0000 UTC m=+35.591909388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls") pod "dns-default-pfdnx" (UID: "44c73487-26c9-4984-a178-1b20cf791453") : secret "dns-default-metrics-tls" not found
Jul 24 18:40:35.717107 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.716651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-ca-trust-extracted\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.717107 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.717059    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-certificates\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.717575 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.717543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-trusted-ca\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.720333 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.720308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-installation-pull-secrets\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.720438 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.720331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99213b47-96d3-4c66-8007-33c8b2fe15d7-cert\") pod \"ingress-canary-b2cwr\" (UID: \"99213b47-96d3-4c66-8007-33c8b2fe15d7\") " pod="openshift-ingress-canary/ingress-canary-b2cwr"
Jul 24 18:40:35.720438 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.720313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-image-registry-private-configuration\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.724206 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.724177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzrmm\" (UniqueName: \"kubernetes.io/projected/44c73487-26c9-4984-a178-1b20cf791453-kube-api-access-pzrmm\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:35.724552 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.724521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-bound-sa-token\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.724672 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.724649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqmqk\" (UniqueName: \"kubernetes.io/projected/99213b47-96d3-4c66-8007-33c8b2fe15d7-kube-api-access-rqmqk\") pod \"ingress-canary-b2cwr\" (UID: \"99213b47-96d3-4c66-8007-33c8b2fe15d7\") " pod="openshift-ingress-canary/ingress-canary-b2cwr"
Jul 24 18:40:35.725535 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.725515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t277\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-kube-api-access-2t277\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:35.816551 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:35.816516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b2cwr"
Jul 24 18:40:36.007803 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:36.007709    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b2cwr"]
Jul 24 18:40:36.191359 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:36.191322    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:40:36.191557 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:36.191322    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:40:36.191557 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:36.191322    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:40:36.194014 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:36.193991    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 18:40:36.194148 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:36.194093    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f8czh\""
Jul 24 18:40:36.194148 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:36.194105    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 18:40:36.194270 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:36.194242    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 18:40:36.194354 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:36.194334    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bhdg2\""
Jul 24 18:40:36.194448 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:36.194427    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 18:40:36.219020 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:36.219000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:36.219116 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:36.219026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:36.219181 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:36.219152    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 18:40:36.219181 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:36.219158    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 18:40:36.219181 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:36.219169    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f59749bb5-kfnt9: secret "image-registry-tls" not found
Jul 24 18:40:36.219326 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:36.219213    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls podName:44c73487-26c9-4984-a178-1b20cf791453 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:37.219197246 +0000 UTC m=+36.594856550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls") pod "dns-default-pfdnx" (UID: "44c73487-26c9-4984-a178-1b20cf791453") : secret "dns-default-metrics-tls" not found
Jul 24 18:40:36.219326 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:36.219231    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls podName:c1cd83d8-1c8d-468c-9ea2-8661cf2eecac nodeName:}" failed. No retries permitted until 2026-07-24 18:40:37.2192221 +0000 UTC m=+36.594881376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls") pod "image-registry-5f59749bb5-kfnt9" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac") : secret "image-registry-tls" not found
Jul 24 18:40:37.226560 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:37.226525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:37.226560 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:37.226559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:37.227030 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:37.226681    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 18:40:37.227030 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:37.226695    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f59749bb5-kfnt9: secret "image-registry-tls" not found
Jul 24 18:40:37.227030 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:37.226708    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 18:40:37.227030 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:37.226751    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls podName:c1cd83d8-1c8d-468c-9ea2-8661cf2eecac nodeName:}" failed. No retries permitted until 2026-07-24 18:40:39.22673547 +0000 UTC m=+38.602394756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls") pod "image-registry-5f59749bb5-kfnt9" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac") : secret "image-registry-tls" not found
Jul 24 18:40:37.227030 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:37.226786    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls podName:44c73487-26c9-4984-a178-1b20cf791453 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:39.226766591 +0000 UTC m=+38.602425864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls") pod "dns-default-pfdnx" (UID: "44c73487-26c9-4984-a178-1b20cf791453") : secret "dns-default-metrics-tls" not found
Jul 24 18:40:37.530396 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:40:37.530315    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99213b47_96d3_4c66_8007_33c8b2fe15d7.slice/crio-a74c93462c3978e13c357f247acbc045e5c2eb7870f4db56b5bb97b56abb9e85 WatchSource:0}: Error finding container a74c93462c3978e13c357f247acbc045e5c2eb7870f4db56b5bb97b56abb9e85: Status 404 returned error can't find the container with id a74c93462c3978e13c357f247acbc045e5c2eb7870f4db56b5bb97b56abb9e85
Jul 24 18:40:38.341158 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:38.341126    2580 generic.go:356] "Generic (PLEG): container finished" podID="1455f583-c317-4edc-98d3-9c8a7f14d37b" containerID="96d725dcb03a5eaeda68923e5d0dc2645c5911e45dc08f67f6db9e7181678467" exitCode=0
Jul 24 18:40:38.341640 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:38.341222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9wq59" event={"ID":"1455f583-c317-4edc-98d3-9c8a7f14d37b","Type":"ContainerDied","Data":"96d725dcb03a5eaeda68923e5d0dc2645c5911e45dc08f67f6db9e7181678467"}
Jul 24 18:40:38.342417 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:38.342368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b2cwr" event={"ID":"99213b47-96d3-4c66-8007-33c8b2fe15d7","Type":"ContainerStarted","Data":"a74c93462c3978e13c357f247acbc045e5c2eb7870f4db56b5bb97b56abb9e85"}
Jul 24 18:40:39.243982 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:39.243763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:39.244133 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:39.243996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:39.244133 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:39.243909    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 18:40:39.244133 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:39.244098    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 18:40:39.244133 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:39.244110    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f59749bb5-kfnt9: secret "image-registry-tls" not found
Jul 24 18:40:39.244133 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:39.244116    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls podName:44c73487-26c9-4984-a178-1b20cf791453 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:43.244097418 +0000 UTC m=+42.619756708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls") pod "dns-default-pfdnx" (UID: "44c73487-26c9-4984-a178-1b20cf791453") : secret "dns-default-metrics-tls" not found
Jul 24 18:40:39.244294 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:39.244142    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls podName:c1cd83d8-1c8d-468c-9ea2-8661cf2eecac nodeName:}" failed. No retries permitted until 2026-07-24 18:40:43.244132944 +0000 UTC m=+42.619792218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls") pod "image-registry-5f59749bb5-kfnt9" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac") : secret "image-registry-tls" not found
Jul 24 18:40:39.348760 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:39.348732    2580 generic.go:356] "Generic (PLEG): container finished" podID="1455f583-c317-4edc-98d3-9c8a7f14d37b" containerID="60308c62d0a9b02f99cd166505c9da525dffd15d5c0977a0027e8e4d8e44ed82" exitCode=0
Jul 24 18:40:39.349156 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:39.348793    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9wq59" event={"ID":"1455f583-c317-4edc-98d3-9c8a7f14d37b","Type":"ContainerDied","Data":"60308c62d0a9b02f99cd166505c9da525dffd15d5c0977a0027e8e4d8e44ed82"}
Jul 24 18:40:40.353212 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:40.353177    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b2cwr" event={"ID":"99213b47-96d3-4c66-8007-33c8b2fe15d7","Type":"ContainerStarted","Data":"5269f832a0a14d17379199ed1309686b38d1785da9d15322525e574da0e4b1c5"}
Jul 24 18:40:40.356866 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:40.356838    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9wq59" event={"ID":"1455f583-c317-4edc-98d3-9c8a7f14d37b","Type":"ContainerStarted","Data":"776eed588a7f61bd68d2dc60e5b3a149028afe22fe9fa641c61dd19282a4112b"}
Jul 24 18:40:40.368351 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:40.368311    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b2cwr" podStartSLOduration=2.7552196589999998 podStartE2EDuration="5.368297928s" totalImagesPullingTime="2.613078269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:40:40.368036983 +0000 UTC m=+39.743696283" watchObservedRunningTime="2026-07-24 18:40:40.368297928 +0000 UTC m=+39.743957223"
Jul 24 18:40:40.388314 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:40.388268    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9wq59" podStartSLOduration=8.028256525 podStartE2EDuration="39.388254192s" totalImagesPullingTime="31.359997667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:40:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:40:40.387492814 +0000 UTC m=+39.763152109" watchObservedRunningTime="2026-07-24 18:40:40.388254192 +0000 UTC m=+39.763913489"
Jul 24 18:40:43.272330 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:43.272295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:43.272330 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:43.272328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:43.272888 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:43.272472    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 18:40:43.272888 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:43.272480    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 18:40:43.272888 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:43.272555    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls podName:44c73487-26c9-4984-a178-1b20cf791453 nodeName:}" failed. No retries permitted until 2026-07-24 18:40:51.272535169 +0000 UTC m=+50.648194468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls") pod "dns-default-pfdnx" (UID: "44c73487-26c9-4984-a178-1b20cf791453") : secret "dns-default-metrics-tls" not found
Jul 24 18:40:43.272888 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:43.272487    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f59749bb5-kfnt9: secret "image-registry-tls" not found
Jul 24 18:40:43.272888 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:43.272616    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls podName:c1cd83d8-1c8d-468c-9ea2-8661cf2eecac nodeName:}" failed. No retries permitted until 2026-07-24 18:40:51.27260377 +0000 UTC m=+50.648263047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls") pod "image-registry-5f59749bb5-kfnt9" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac") : secret "image-registry-tls" not found
Jul 24 18:40:51.326979 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:51.326931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:40:51.326979 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:40:51.326977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:40:51.327533 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:51.327066    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 18:40:51.327533 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:51.327067    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 18:40:51.327533 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:51.327077    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f59749bb5-kfnt9: secret "image-registry-tls" not found
Jul 24 18:40:51.327533 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:51.327127    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls podName:44c73487-26c9-4984-a178-1b20cf791453 nodeName:}" failed. No retries permitted until 2026-07-24 18:41:07.327110797 +0000 UTC m=+66.702770081 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls") pod "dns-default-pfdnx" (UID: "44c73487-26c9-4984-a178-1b20cf791453") : secret "dns-default-metrics-tls" not found
Jul 24 18:40:51.327533 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:40:51.327141    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls podName:c1cd83d8-1c8d-468c-9ea2-8661cf2eecac nodeName:}" failed. No retries permitted until 2026-07-24 18:41:07.327135013 +0000 UTC m=+66.702794286 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls") pod "image-registry-5f59749bb5-kfnt9" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac") : secret "image-registry-tls" not found
Jul 24 18:41:00.895156 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.895119    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts"]
Jul 24 18:41:00.900586 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.897668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts"
Jul 24 18:41:00.904147 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.904123    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 24 18:41:00.904277 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.904156    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 24 18:41:00.904277 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.904187    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 24 18:41:00.904277 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.904124    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 24 18:41:00.904737 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.904723    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7ldcc\""
Jul 24 18:41:00.914079 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.914060    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts"]
Jul 24 18:41:00.991411 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.991367    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-d65zr"]
Jul 24 18:41:00.992976 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.992962    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-d65zr"
Jul 24 18:41:00.995232 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.995217    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-h6r8k\""
Jul 24 18:41:00.995514 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.995495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp7fg\" (UniqueName: \"kubernetes.io/projected/7fd19688-7aa4-4567-8ada-87afa3c0a7a9-kube-api-access-tp7fg\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts\" (UID: \"7fd19688-7aa4-4567-8ada-87afa3c0a7a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts"
Jul 24 18:41:00.995588 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.995548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fd19688-7aa4-4567-8ada-87afa3c0a7a9-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts\" (UID: \"7fd19688-7aa4-4567-8ada-87afa3c0a7a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts"
Jul 24 18:41:00.995634 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.995589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7fd19688-7aa4-4567-8ada-87afa3c0a7a9-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts\" (UID: \"7fd19688-7aa4-4567-8ada-87afa3c0a7a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts"
Jul 24 18:41:00.998658 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:00.998639    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-79qbp"]
Jul 24 18:41:01.000535 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.000511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:01.002609 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.002592    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 24 18:41:01.002752 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.002739    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 24 18:41:01.002824 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.002752    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 24 18:41:01.002824 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.002787    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 24 18:41:01.003040 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.003025    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-q2nzg\""
Jul 24 18:41:01.007600 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.007579    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-d65zr"]
Jul 24 18:41:01.008934 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.008915    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 24 18:41:01.020297 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.020275    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-79qbp"]
Jul 24 18:41:01.096572 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.096547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fd19688-7aa4-4567-8ada-87afa3c0a7a9-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts\" (UID: \"7fd19688-7aa4-4567-8ada-87afa3c0a7a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts"
Jul 24 18:41:01.096699 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.096591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/788bff11-3f28-40aa-b52d-caebf1949d75-trusted-ca\") pod \"console-operator-584dcb6b54-79qbp\" (UID: \"788bff11-3f28-40aa-b52d-caebf1949d75\") " pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:01.096699 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.096624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7fd19688-7aa4-4567-8ada-87afa3c0a7a9-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts\" (UID: \"7fd19688-7aa4-4567-8ada-87afa3c0a7a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts"
Jul 24 18:41:01.096699 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.096655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsbrs\" (UniqueName: \"kubernetes.io/projected/c9b4b390-c70e-49fc-a723-38260b9bc7ac-kube-api-access-fsbrs\") pod \"network-check-source-7d88cb55d8-d65zr\" (UID: \"c9b4b390-c70e-49fc-a723-38260b9bc7ac\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-d65zr"
Jul 24 18:41:01.096825 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.096733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxnmf\" (UniqueName: \"kubernetes.io/projected/788bff11-3f28-40aa-b52d-caebf1949d75-kube-api-access-gxnmf\") pod \"console-operator-584dcb6b54-79qbp\" (UID: \"788bff11-3f28-40aa-b52d-caebf1949d75\") " pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:01.096825 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.096776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/788bff11-3f28-40aa-b52d-caebf1949d75-config\") pod \"console-operator-584dcb6b54-79qbp\" (UID: \"788bff11-3f28-40aa-b52d-caebf1949d75\") " pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:01.096900 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.096840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/788bff11-3f28-40aa-b52d-caebf1949d75-serving-cert\") pod \"console-operator-584dcb6b54-79qbp\" (UID: \"788bff11-3f28-40aa-b52d-caebf1949d75\") " pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:01.096900 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.096877    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp7fg\" (UniqueName: \"kubernetes.io/projected/7fd19688-7aa4-4567-8ada-87afa3c0a7a9-kube-api-access-tp7fg\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts\" (UID: \"7fd19688-7aa4-4567-8ada-87afa3c0a7a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts"
Jul 24 18:41:01.098766 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.098746    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 24 18:41:01.098900 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.098881    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 24 18:41:01.103871 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.103853    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 24 18:41:01.107530 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.107513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7fd19688-7aa4-4567-8ada-87afa3c0a7a9-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts\" (UID: \"7fd19688-7aa4-4567-8ada-87afa3c0a7a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts"
Jul 24 18:41:01.110466 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.110451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fd19688-7aa4-4567-8ada-87afa3c0a7a9-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts\" (UID: \"7fd19688-7aa4-4567-8ada-87afa3c0a7a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts"
Jul 24 18:41:01.114027 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.114007    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 24 18:41:01.124542 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.124521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp7fg\" (UniqueName: \"kubernetes.io/projected/7fd19688-7aa4-4567-8ada-87afa3c0a7a9-kube-api-access-tp7fg\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts\" (UID: \"7fd19688-7aa4-4567-8ada-87afa3c0a7a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts"
Jul 24 18:41:01.197477 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.197417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/788bff11-3f28-40aa-b52d-caebf1949d75-serving-cert\") pod \"console-operator-584dcb6b54-79qbp\" (UID: \"788bff11-3f28-40aa-b52d-caebf1949d75\") " pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:01.197568 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.197493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/788bff11-3f28-40aa-b52d-caebf1949d75-trusted-ca\") pod \"console-operator-584dcb6b54-79qbp\" (UID: \"788bff11-3f28-40aa-b52d-caebf1949d75\") " pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:01.197568 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.197532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsbrs\" (UniqueName: \"kubernetes.io/projected/c9b4b390-c70e-49fc-a723-38260b9bc7ac-kube-api-access-fsbrs\") pod \"network-check-source-7d88cb55d8-d65zr\" (UID: \"c9b4b390-c70e-49fc-a723-38260b9bc7ac\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-d65zr"
Jul 24 18:41:01.197635 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.197574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxnmf\" (UniqueName: \"kubernetes.io/projected/788bff11-3f28-40aa-b52d-caebf1949d75-kube-api-access-gxnmf\") pod \"console-operator-584dcb6b54-79qbp\" (UID: \"788bff11-3f28-40aa-b52d-caebf1949d75\") " pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:01.197635 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.197610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/788bff11-3f28-40aa-b52d-caebf1949d75-config\") pod \"console-operator-584dcb6b54-79qbp\" (UID: \"788bff11-3f28-40aa-b52d-caebf1949d75\") " pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:01.198278 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.198261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/788bff11-3f28-40aa-b52d-caebf1949d75-trusted-ca\") pod \"console-operator-584dcb6b54-79qbp\" (UID: \"788bff11-3f28-40aa-b52d-caebf1949d75\") " pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:01.199648 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.199628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 24 18:41:01.199737 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.199660    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 24 18:41:01.204560 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.204543    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 18:41:01.204645 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.204619    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 24 18:41:01.208530 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.208507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/788bff11-3f28-40aa-b52d-caebf1949d75-config\") pod \"console-operator-584dcb6b54-79qbp\" (UID: \"788bff11-3f28-40aa-b52d-caebf1949d75\") " pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:01.209840 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.209817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/788bff11-3f28-40aa-b52d-caebf1949d75-serving-cert\") pod \"console-operator-584dcb6b54-79qbp\" (UID: \"788bff11-3f28-40aa-b52d-caebf1949d75\") " pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:01.210881 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.210867    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7ldcc\""
Jul 24 18:41:01.214653 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.214632    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 18:41:01.214653 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.214646    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 24 18:41:01.220260 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.220246    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts"
Jul 24 18:41:01.225857 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.225836    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxnmf\" (UniqueName: \"kubernetes.io/projected/788bff11-3f28-40aa-b52d-caebf1949d75-kube-api-access-gxnmf\") pod \"console-operator-584dcb6b54-79qbp\" (UID: \"788bff11-3f28-40aa-b52d-caebf1949d75\") " pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:01.225944 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.225853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsbrs\" (UniqueName: \"kubernetes.io/projected/c9b4b390-c70e-49fc-a723-38260b9bc7ac-kube-api-access-fsbrs\") pod \"network-check-source-7d88cb55d8-d65zr\" (UID: \"c9b4b390-c70e-49fc-a723-38260b9bc7ac\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-d65zr"
Jul 24 18:41:01.303311 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.303288    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-h6r8k\""
Jul 24 18:41:01.311108 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.311079    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-q2nzg\""
Jul 24 18:41:01.312124 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.312102    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-d65zr"
Jul 24 18:41:01.319695 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.319675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:01.342174 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.342110    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts"]
Jul 24 18:41:01.354024 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:01.347917    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd19688_7aa4_4567_8ada_87afa3c0a7a9.slice/crio-d60d9f7dbb373aa8826495fa5281afd2a055e2c3c6f98c2fd63b19216182ae7f WatchSource:0}: Error finding container d60d9f7dbb373aa8826495fa5281afd2a055e2c3c6f98c2fd63b19216182ae7f: Status 404 returned error can't find the container with id d60d9f7dbb373aa8826495fa5281afd2a055e2c3c6f98c2fd63b19216182ae7f
Jul 24 18:41:01.406963 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.406923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts" event={"ID":"7fd19688-7aa4-4567-8ada-87afa3c0a7a9","Type":"ContainerStarted","Data":"d60d9f7dbb373aa8826495fa5281afd2a055e2c3c6f98c2fd63b19216182ae7f"}
Jul 24 18:41:01.452307 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.452242    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-d65zr"]
Jul 24 18:41:01.454750 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:01.454723    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b4b390_c70e_49fc_a723_38260b9bc7ac.slice/crio-54c739e27ad79d1247c36faa8308a3ae6de0f932589eea0257c044598483325c WatchSource:0}: Error finding container 54c739e27ad79d1247c36faa8308a3ae6de0f932589eea0257c044598483325c: Status 404 returned error can't find the container with id 54c739e27ad79d1247c36faa8308a3ae6de0f932589eea0257c044598483325c
Jul 24 18:41:01.466770 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:01.466748    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-79qbp"]
Jul 24 18:41:01.469463 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:01.469444    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788bff11_3f28_40aa_b52d_caebf1949d75.slice/crio-95107da7361feaed1b69eef75adf7cfea3bf048edcee9e1b8dbc55b6b01bc387 WatchSource:0}: Error finding container 95107da7361feaed1b69eef75adf7cfea3bf048edcee9e1b8dbc55b6b01bc387: Status 404 returned error can't find the container with id 95107da7361feaed1b69eef75adf7cfea3bf048edcee9e1b8dbc55b6b01bc387
Jul 24 18:41:02.333347 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:02.333307    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hrxnr"
Jul 24 18:41:02.412244 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:02.412211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-d65zr" event={"ID":"c9b4b390-c70e-49fc-a723-38260b9bc7ac","Type":"ContainerStarted","Data":"54c739e27ad79d1247c36faa8308a3ae6de0f932589eea0257c044598483325c"}
Jul 24 18:41:02.414025 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:02.413995    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-79qbp" event={"ID":"788bff11-3f28-40aa-b52d-caebf1949d75","Type":"ContainerStarted","Data":"95107da7361feaed1b69eef75adf7cfea3bf048edcee9e1b8dbc55b6b01bc387"}
Jul 24 18:41:05.422797 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.422763    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/0.log"
Jul 24 18:41:05.423227 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.422809    2580 generic.go:356] "Generic (PLEG): container finished" podID="788bff11-3f28-40aa-b52d-caebf1949d75" containerID="67443131b24b5f9362c02d33fb03309e9ec875bff0f35e633128b96fba22c781" exitCode=255
Jul 24 18:41:05.423227 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.422904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-79qbp" event={"ID":"788bff11-3f28-40aa-b52d-caebf1949d75","Type":"ContainerDied","Data":"67443131b24b5f9362c02d33fb03309e9ec875bff0f35e633128b96fba22c781"}
Jul 24 18:41:05.423227 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.423073    2580 scope.go:117] "RemoveContainer" containerID="67443131b24b5f9362c02d33fb03309e9ec875bff0f35e633128b96fba22c781"
Jul 24 18:41:05.424393 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.424357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-d65zr" event={"ID":"c9b4b390-c70e-49fc-a723-38260b9bc7ac","Type":"ContainerStarted","Data":"9c7186446b552099c6bcd249d2ce70867889e3d8ca0dc9c640a09ba7de54a1f5"}
Jul 24 18:41:05.425796 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.425765    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts" event={"ID":"7fd19688-7aa4-4567-8ada-87afa3c0a7a9","Type":"ContainerStarted","Data":"0c1825b964188592e69e97e0c7455cb4920e859b5a25019a268dee980bd11cfc"}
Jul 24 18:41:05.454705 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.454667    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-d65zr" podStartSLOduration=1.918381966 podStartE2EDuration="5.454654101s" totalImagesPullingTime="3.536272135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:05.454001325 +0000 UTC m=+64.829660624" watchObservedRunningTime="2026-07-24 18:41:05.454654101 +0000 UTC m=+64.830313397"
Jul 24 18:41:05.468303 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.468259    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts" podStartSLOduration=1.837417754 podStartE2EDuration="5.468241918s" totalImagesPullingTime="3.630824164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:05.467679116 +0000 UTC m=+64.843338411" watchObservedRunningTime="2026-07-24 18:41:05.468241918 +0000 UTC m=+64.843901216"
Jul 24 18:41:05.778375 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.778308    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k276f"]
Jul 24 18:41:05.782680 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.782659    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:05.785212 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.785189    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 18:41:05.785314 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.785284    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 18:41:05.785531 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.785515    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 18:41:05.785773 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.785756    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 18:41:05.787726 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.787708    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2bkk8\""
Jul 24 18:41:05.804410 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.804367    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k276f"]
Jul 24 18:41:05.834950 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.834928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:05.835070 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.834962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04e6b58a-4a62-4635-9058-0899b5799cfe-data-volume\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:05.835070 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.834993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04e6b58a-4a62-4635-9058-0899b5799cfe-crio-socket\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:05.835147 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.835078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvjx9\" (UniqueName: \"kubernetes.io/projected/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-api-access-kvjx9\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:05.835147 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.835123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:05.935620 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.935592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04e6b58a-4a62-4635-9058-0899b5799cfe-crio-socket\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:05.935778 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.935689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvjx9\" (UniqueName: \"kubernetes.io/projected/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-api-access-kvjx9\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:05.935778 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.935716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:05.935778 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.935711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04e6b58a-4a62-4635-9058-0899b5799cfe-crio-socket\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:05.935940 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.935802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:05.935940 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:05.935833    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 18:41:05.935940 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:05.935899    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm podName:04e6b58a-4a62-4635-9058-0899b5799cfe nodeName:}" failed. No retries permitted until 2026-07-24 18:41:06.435879932 +0000 UTC m=+65.811539225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k276f" (UID: "04e6b58a-4a62-4635-9058-0899b5799cfe") : configmap "kube-rbac-proxy" not found
Jul 24 18:41:05.935940 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:05.935921    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 18:41:05.936166 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:05.935991    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls podName:04e6b58a-4a62-4635-9058-0899b5799cfe nodeName:}" failed. No retries permitted until 2026-07-24 18:41:06.43597235 +0000 UTC m=+65.811631644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k276f" (UID: "04e6b58a-4a62-4635-9058-0899b5799cfe") : secret "insights-runtime-extractor-tls" not found
Jul 24 18:41:05.936166 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.935836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04e6b58a-4a62-4635-9058-0899b5799cfe-data-volume\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:05.936166 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.936145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04e6b58a-4a62-4635-9058-0899b5799cfe-data-volume\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:05.944079 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:05.944061    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvjx9\" (UniqueName: \"kubernetes.io/projected/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-api-access-kvjx9\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:06.058359 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.058285    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-tnw9x"]
Jul 24 18:41:06.062578 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.062554    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tnw9x"
Jul 24 18:41:06.064884 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.064854    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 24 18:41:06.065012 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.064854    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 24 18:41:06.065012 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.064931    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hkw5q\""
Jul 24 18:41:06.075228 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.075194    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-tnw9x"]
Jul 24 18:41:06.137450 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.137422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lpks\" (UniqueName: \"kubernetes.io/projected/600033b2-d859-44d2-9c67-5f16e8ce6ad0-kube-api-access-4lpks\") pod \"migrator-6865c9967-tnw9x\" (UID: \"600033b2-d859-44d2-9c67-5f16e8ce6ad0\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tnw9x"
Jul 24 18:41:06.238090 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.238063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lpks\" (UniqueName: \"kubernetes.io/projected/600033b2-d859-44d2-9c67-5f16e8ce6ad0-kube-api-access-4lpks\") pod \"migrator-6865c9967-tnw9x\" (UID: \"600033b2-d859-44d2-9c67-5f16e8ce6ad0\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tnw9x"
Jul 24 18:41:06.246074 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.246056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lpks\" (UniqueName: \"kubernetes.io/projected/600033b2-d859-44d2-9c67-5f16e8ce6ad0-kube-api-access-4lpks\") pod \"migrator-6865c9967-tnw9x\" (UID: \"600033b2-d859-44d2-9c67-5f16e8ce6ad0\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tnw9x"
Jul 24 18:41:06.370783 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.370744    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tnw9x"
Jul 24 18:41:06.431527 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.431496    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 18:41:06.433008 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.432984    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/0.log"
Jul 24 18:41:06.433106 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.433037    2580 generic.go:356] "Generic (PLEG): container finished" podID="788bff11-3f28-40aa-b52d-caebf1949d75" containerID="aa7fedec3b4ede2b90d503d8a1b1a982ff6b06ac093f01cd73cef7ce434c4431" exitCode=255
Jul 24 18:41:06.433148 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.433131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-79qbp" event={"ID":"788bff11-3f28-40aa-b52d-caebf1949d75","Type":"ContainerDied","Data":"aa7fedec3b4ede2b90d503d8a1b1a982ff6b06ac093f01cd73cef7ce434c4431"}
Jul 24 18:41:06.433212 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.433196    2580 scope.go:117] "RemoveContainer" containerID="67443131b24b5f9362c02d33fb03309e9ec875bff0f35e633128b96fba22c781"
Jul 24 18:41:06.433490 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.433430    2580 scope.go:117] "RemoveContainer" containerID="aa7fedec3b4ede2b90d503d8a1b1a982ff6b06ac093f01cd73cef7ce434c4431"
Jul 24 18:41:06.433659 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:06.433635    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-79qbp_openshift-console-operator(788bff11-3f28-40aa-b52d-caebf1949d75)\"" pod="openshift-console-operator/console-operator-584dcb6b54-79qbp" podUID="788bff11-3f28-40aa-b52d-caebf1949d75"
Jul 24 18:41:06.440026 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.440006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:06.440137 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.440070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:06.440199 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:06.440160    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 18:41:06.440199 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:06.440175    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 18:41:06.440303 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:06.440208    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm podName:04e6b58a-4a62-4635-9058-0899b5799cfe nodeName:}" failed. No retries permitted until 2026-07-24 18:41:07.440193562 +0000 UTC m=+66.815852855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k276f" (UID: "04e6b58a-4a62-4635-9058-0899b5799cfe") : configmap "kube-rbac-proxy" not found
Jul 24 18:41:06.440303 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:06.440253    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls podName:04e6b58a-4a62-4635-9058-0899b5799cfe nodeName:}" failed. No retries permitted until 2026-07-24 18:41:07.440235964 +0000 UTC m=+66.815895249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k276f" (UID: "04e6b58a-4a62-4635-9058-0899b5799cfe") : secret "insights-runtime-extractor-tls" not found
Jul 24 18:41:06.489880 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.489851    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-tnw9x"]
Jul 24 18:41:06.493239 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:06.493210    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600033b2_d859_44d2_9c67_5f16e8ce6ad0.slice/crio-98bfae2c396490e99e48a478f48c4ca6576cd59173314f370bd6311ccd627a65 WatchSource:0}: Error finding container 98bfae2c396490e99e48a478f48c4ca6576cd59173314f370bd6311ccd627a65: Status 404 returned error can't find the container with id 98bfae2c396490e99e48a478f48c4ca6576cd59173314f370bd6311ccd627a65
Jul 24 18:41:06.843820 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.843789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs\") pod \"network-metrics-daemon-xkxwb\" (UID: \"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0\") " pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:41:06.845600 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.845579    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 18:41:06.854046 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:06.854028    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 18:41:06.854118 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:06.854077    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs podName:2e0785e8-c533-44cb-bc6f-d11ed1bb60f0 nodeName:}" failed. No retries permitted until 2026-07-24 18:42:10.854064213 +0000 UTC m=+130.229723500 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs") pod "network-metrics-daemon-xkxwb" (UID: "2e0785e8-c533-44cb-bc6f-d11ed1bb60f0") : secret "metrics-daemon-secret" not found
Jul 24 18:41:06.944849 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.944823    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb9t\" (UniqueName: \"kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t\") pod \"network-check-target-f6w5m\" (UID: \"d932da4a-af42-4d08-a40d-a1c2ce4e1123\") " pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:41:06.945022 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.945007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:41:06.947062 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.947040    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 18:41:06.947242 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.947223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sb9t\" (UniqueName: \"kubernetes.io/projected/d932da4a-af42-4d08-a40d-a1c2ce4e1123-kube-api-access-2sb9t\") pod \"network-check-target-f6w5m\" (UID: \"d932da4a-af42-4d08-a40d-a1c2ce4e1123\") " pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:41:06.957582 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:06.957557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc4eada-1823-4bc2-add7-98ef884c80d4-original-pull-secret\") pod \"global-pull-secret-syncer-vn9rh\" (UID: \"8bc4eada-1823-4bc2-add7-98ef884c80d4\") " pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:41:07.104597 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.104530    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bhdg2\""
Jul 24 18:41:07.113368 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.113342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:41:07.114606 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.114581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vn9rh"
Jul 24 18:41:07.252804 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.252769    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f6w5m"]
Jul 24 18:41:07.261470 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:07.259105    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd932da4a_af42_4d08_a40d_a1c2ce4e1123.slice/crio-6e6b6aa4b63c62d3844dbe69faf41201a5ac6701f3d74d6b0027fd86e8abaf78 WatchSource:0}: Error finding container 6e6b6aa4b63c62d3844dbe69faf41201a5ac6701f3d74d6b0027fd86e8abaf78: Status 404 returned error can't find the container with id 6e6b6aa4b63c62d3844dbe69faf41201a5ac6701f3d74d6b0027fd86e8abaf78
Jul 24 18:41:07.271410 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.271315    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vn9rh"]
Jul 24 18:41:07.274405 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:07.274361    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc4eada_1823_4bc2_add7_98ef884c80d4.slice/crio-dd8dc90820e9f500ae2cece79d2e555904dc4bc368c2680233a40ed47a293731 WatchSource:0}: Error finding container dd8dc90820e9f500ae2cece79d2e555904dc4bc368c2680233a40ed47a293731: Status 404 returned error can't find the container with id dd8dc90820e9f500ae2cece79d2e555904dc4bc368c2680233a40ed47a293731
Jul 24 18:41:07.348094 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.348067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:41:07.348190 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.348099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:41:07.348230 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:07.348222    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 18:41:07.348260 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:07.348238    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f59749bb5-kfnt9: secret "image-registry-tls" not found
Jul 24 18:41:07.348290 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:07.348274    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls podName:c1cd83d8-1c8d-468c-9ea2-8661cf2eecac nodeName:}" failed. No retries permitted until 2026-07-24 18:41:39.348262699 +0000 UTC m=+98.723921973 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls") pod "image-registry-5f59749bb5-kfnt9" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac") : secret "image-registry-tls" not found
Jul 24 18:41:07.348326 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:07.348221    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 18:41:07.348366 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:07.348341    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls podName:44c73487-26c9-4984-a178-1b20cf791453 nodeName:}" failed. No retries permitted until 2026-07-24 18:41:39.348327329 +0000 UTC m=+98.723986603 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls") pod "dns-default-pfdnx" (UID: "44c73487-26c9-4984-a178-1b20cf791453") : secret "dns-default-metrics-tls" not found
Jul 24 18:41:07.437474 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.437398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tnw9x" event={"ID":"600033b2-d859-44d2-9c67-5f16e8ce6ad0","Type":"ContainerStarted","Data":"98bfae2c396490e99e48a478f48c4ca6576cd59173314f370bd6311ccd627a65"}
Jul 24 18:41:07.438868 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.438848    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 18:41:07.439172 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.439155    2580 scope.go:117] "RemoveContainer" containerID="aa7fedec3b4ede2b90d503d8a1b1a982ff6b06ac093f01cd73cef7ce434c4431"
Jul 24 18:41:07.439340 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:07.439319    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-79qbp_openshift-console-operator(788bff11-3f28-40aa-b52d-caebf1949d75)\"" pod="openshift-console-operator/console-operator-584dcb6b54-79qbp" podUID="788bff11-3f28-40aa-b52d-caebf1949d75"
Jul 24 18:41:07.440506 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.440482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f6w5m" event={"ID":"d932da4a-af42-4d08-a40d-a1c2ce4e1123","Type":"ContainerStarted","Data":"b09972e44d28214571858019a938a7bb36ceb180fcf5a0fa98b67ba8031d5fdc"}
Jul 24 18:41:07.440606 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.440528    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f6w5m" event={"ID":"d932da4a-af42-4d08-a40d-a1c2ce4e1123","Type":"ContainerStarted","Data":"6e6b6aa4b63c62d3844dbe69faf41201a5ac6701f3d74d6b0027fd86e8abaf78"}
Jul 24 18:41:07.440606 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.440586    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:41:07.441478 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.441455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vn9rh" event={"ID":"8bc4eada-1823-4bc2-add7-98ef884c80d4","Type":"ContainerStarted","Data":"dd8dc90820e9f500ae2cece79d2e555904dc4bc368c2680233a40ed47a293731"}
Jul 24 18:41:07.449394 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.449363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:07.449476 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.449460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:07.449519 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:07.449491    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 18:41:07.449550 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:07.449542    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 18:41:07.449583 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:07.449561    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls podName:04e6b58a-4a62-4635-9058-0899b5799cfe nodeName:}" failed. No retries permitted until 2026-07-24 18:41:09.449540993 +0000 UTC m=+68.825200285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k276f" (UID: "04e6b58a-4a62-4635-9058-0899b5799cfe") : secret "insights-runtime-extractor-tls" not found
Jul 24 18:41:07.449583 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:07.449581    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm podName:04e6b58a-4a62-4635-9058-0899b5799cfe nodeName:}" failed. No retries permitted until 2026-07-24 18:41:09.449569684 +0000 UTC m=+68.825228959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k276f" (UID: "04e6b58a-4a62-4635-9058-0899b5799cfe") : configmap "kube-rbac-proxy" not found
Jul 24 18:41:07.474676 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.474559    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f6w5m" podStartSLOduration=66.474543274 podStartE2EDuration="1m6.474543274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:07.473896122 +0000 UTC m=+66.849555417" watchObservedRunningTime="2026-07-24 18:41:07.474543274 +0000 UTC m=+66.850202571"
Jul 24 18:41:07.967944 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:07.967876    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l9drv_28bdcc77-a763-49e4-abb3-98de9bd1d399/dns-node-resolver/0.log"
Jul 24 18:41:08.447336 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:08.447291    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tnw9x" event={"ID":"600033b2-d859-44d2-9c67-5f16e8ce6ad0","Type":"ContainerStarted","Data":"af34c4586b7fec3de5db5bf97fb8f89536ffadf509569687140d6e13e43c4286"}
Jul 24 18:41:08.447809 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:08.447368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tnw9x" event={"ID":"600033b2-d859-44d2-9c67-5f16e8ce6ad0","Type":"ContainerStarted","Data":"c11878dca92f76030baf28c37214c06151eb1ab7c7ea6111f66438ec69f9b268"}
Jul 24 18:41:08.767674 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:08.767597    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9nzfl_2360ee4d-31bb-4de1-91ad-278352f88f22/node-ca/0.log"
Jul 24 18:41:09.367400 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:09.367352    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b2cwr_99213b47-96d3-4c66-8007-33c8b2fe15d7/serve-healthcheck-canary/0.log"
Jul 24 18:41:09.465295 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:09.465249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:09.465748 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:09.465316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:09.465748 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:09.465345    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 18:41:09.465748 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:09.465432    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm podName:04e6b58a-4a62-4635-9058-0899b5799cfe nodeName:}" failed. No retries permitted until 2026-07-24 18:41:13.465409675 +0000 UTC m=+72.841068954 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k276f" (UID: "04e6b58a-4a62-4635-9058-0899b5799cfe") : configmap "kube-rbac-proxy" not found
Jul 24 18:41:09.465748 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:09.465473    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 18:41:09.465748 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:09.465523    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls podName:04e6b58a-4a62-4635-9058-0899b5799cfe nodeName:}" failed. No retries permitted until 2026-07-24 18:41:13.465508503 +0000 UTC m=+72.841167796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k276f" (UID: "04e6b58a-4a62-4635-9058-0899b5799cfe") : secret "insights-runtime-extractor-tls" not found
Jul 24 18:41:11.319948 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:11.319917    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:11.320314 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:11.319968    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:11.320314 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:11.320258    2580 scope.go:117] "RemoveContainer" containerID="aa7fedec3b4ede2b90d503d8a1b1a982ff6b06ac093f01cd73cef7ce434c4431"
Jul 24 18:41:11.320431 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:11.320411    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-79qbp_openshift-console-operator(788bff11-3f28-40aa-b52d-caebf1949d75)\"" pod="openshift-console-operator/console-operator-584dcb6b54-79qbp" podUID="788bff11-3f28-40aa-b52d-caebf1949d75"
Jul 24 18:41:12.461797 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:12.461758    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vn9rh" event={"ID":"8bc4eada-1823-4bc2-add7-98ef884c80d4","Type":"ContainerStarted","Data":"e5fdb2b294d4fdb31b0b75fa298154f4529f215d0870c34132b4d3c434fbae09"}
Jul 24 18:41:12.478589 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:12.478537    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vn9rh" podStartSLOduration=67.222184454 podStartE2EDuration="1m11.478523587s" totalImagesPullingTime="4.256339133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:12.477973141 +0000 UTC m=+71.853632437" watchObservedRunningTime="2026-07-24 18:41:12.478523587 +0000 UTC m=+71.854182930"
Jul 24 18:41:12.478951 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:12.478913    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tnw9x" podStartSLOduration=5.281538623 podStartE2EDuration="6.478902693s" totalImagesPullingTime="1.19736407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:08.469088525 +0000 UTC m=+67.844747822" watchObservedRunningTime="2026-07-24 18:41:12.478902693 +0000 UTC m=+71.854561993"
Jul 24 18:41:13.498187 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:13.498142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:13.498187 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:13.498192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:13.498661 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:13.498257    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 18:41:13.498661 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:13.498287    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 18:41:13.498661 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:13.498315    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm podName:04e6b58a-4a62-4635-9058-0899b5799cfe nodeName:}" failed. No retries permitted until 2026-07-24 18:41:21.498301091 +0000 UTC m=+80.873960364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k276f" (UID: "04e6b58a-4a62-4635-9058-0899b5799cfe") : configmap "kube-rbac-proxy" not found
Jul 24 18:41:13.498661 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:13.498331    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls podName:04e6b58a-4a62-4635-9058-0899b5799cfe nodeName:}" failed. No retries permitted until 2026-07-24 18:41:21.49832316 +0000 UTC m=+80.873982433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k276f" (UID: "04e6b58a-4a62-4635-9058-0899b5799cfe") : secret "insights-runtime-extractor-tls" not found
Jul 24 18:41:21.554981 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:21.554934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:21.555496 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:21.554998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:21.555665 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:21.555641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04e6b58a-4a62-4635-9058-0899b5799cfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:21.557227 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:21.557209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04e6b58a-4a62-4635-9058-0899b5799cfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k276f\" (UID: \"04e6b58a-4a62-4635-9058-0899b5799cfe\") " pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:21.691147 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:21.691124    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k276f"
Jul 24 18:41:21.812352 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:21.812169    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k276f"]
Jul 24 18:41:21.816440 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:21.816410    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e6b58a_4a62_4635_9058_0899b5799cfe.slice/crio-355def2a0772ae2dcc7c28b08b3075f8b6aa831c89602ae30c5de6fdce6fbb9a WatchSource:0}: Error finding container 355def2a0772ae2dcc7c28b08b3075f8b6aa831c89602ae30c5de6fdce6fbb9a: Status 404 returned error can't find the container with id 355def2a0772ae2dcc7c28b08b3075f8b6aa831c89602ae30c5de6fdce6fbb9a
Jul 24 18:41:22.493938 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:22.493842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k276f" event={"ID":"04e6b58a-4a62-4635-9058-0899b5799cfe","Type":"ContainerStarted","Data":"355def2a0772ae2dcc7c28b08b3075f8b6aa831c89602ae30c5de6fdce6fbb9a"}
Jul 24 18:41:25.191186 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:25.191043    2580 scope.go:117] "RemoveContainer" containerID="aa7fedec3b4ede2b90d503d8a1b1a982ff6b06ac093f01cd73cef7ce434c4431"
Jul 24 18:41:27.515404 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:27.514531    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 18:41:27.515404 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:27.514648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-79qbp" event={"ID":"788bff11-3f28-40aa-b52d-caebf1949d75","Type":"ContainerStarted","Data":"336384781e96651eb7573da432dc3c86a52df471fbc847f270662eb8b9e49cb2"}
Jul 24 18:41:27.515404 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:27.514946    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:27.519415 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:27.519086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k276f" event={"ID":"04e6b58a-4a62-4635-9058-0899b5799cfe","Type":"ContainerStarted","Data":"b337e4153159fabae2bf449f3ff3bcc02cbd9f8fb0d4e21cf8a608c117818dfd"}
Jul 24 18:41:27.523434 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:27.522242    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-79qbp"
Jul 24 18:41:27.538716 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:27.538660    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-79qbp" podStartSLOduration=24.02612398 podStartE2EDuration="27.538647014s" totalImagesPullingTime="3.512523034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:27.538119568 +0000 UTC m=+86.913778865" watchObservedRunningTime="2026-07-24 18:41:27.538647014 +0000 UTC m=+86.914306311"
Jul 24 18:41:29.142818 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.142779    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"]
Jul 24 18:41:29.161623 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.161596    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"
Jul 24 18:41:29.167314 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.167238    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 24 18:41:29.167952 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.167930    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 24 18:41:29.169456 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.168955    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 24 18:41:29.169456 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.169263    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 24 18:41:29.183083 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.183026    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"]
Jul 24 18:41:29.217470 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.217436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8e920df2-78b9-46e7-a34c-26f4226d8724-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b88fb6dc8-d594z\" (UID: \"8e920df2-78b9-46e7-a34c-26f4226d8724\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"
Jul 24 18:41:29.217615 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.217506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e920df2-78b9-46e7-a34c-26f4226d8724-tmp\") pod \"klusterlet-addon-workmgr-5b88fb6dc8-d594z\" (UID: \"8e920df2-78b9-46e7-a34c-26f4226d8724\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"
Jul 24 18:41:29.217615 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.217554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkmbh\" (UniqueName: \"kubernetes.io/projected/8e920df2-78b9-46e7-a34c-26f4226d8724-kube-api-access-zkmbh\") pod \"klusterlet-addon-workmgr-5b88fb6dc8-d594z\" (UID: \"8e920df2-78b9-46e7-a34c-26f4226d8724\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"
Jul 24 18:41:29.280074 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.280041    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-kj4n8"]
Jul 24 18:41:29.287993 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.287968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-kj4n8"
Jul 24 18:41:29.288414 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.288390    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-w8s88"]
Jul 24 18:41:29.290344 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.290320    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7drrc\""
Jul 24 18:41:29.290596 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.290573    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 24 18:41:29.290688 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.290639    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 24 18:41:29.293088 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.293068    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-w8s88"
Jul 24 18:41:29.295687 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.295642    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6rnwf\""
Jul 24 18:41:29.295687 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.295658    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 24 18:41:29.295917 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.295746    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 24 18:41:29.300578 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.300549    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-kj4n8"]
Jul 24 18:41:29.305356 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.305310    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-w8s88"]
Jul 24 18:41:29.318148 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.318126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e920df2-78b9-46e7-a34c-26f4226d8724-tmp\") pod \"klusterlet-addon-workmgr-5b88fb6dc8-d594z\" (UID: \"8e920df2-78b9-46e7-a34c-26f4226d8724\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"
Jul 24 18:41:29.318258 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.318171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkmbh\" (UniqueName: \"kubernetes.io/projected/8e920df2-78b9-46e7-a34c-26f4226d8724-kube-api-access-zkmbh\") pod \"klusterlet-addon-workmgr-5b88fb6dc8-d594z\" (UID: \"8e920df2-78b9-46e7-a34c-26f4226d8724\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"
Jul 24 18:41:29.318319 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.318275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8e920df2-78b9-46e7-a34c-26f4226d8724-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b88fb6dc8-d594z\" (UID: \"8e920df2-78b9-46e7-a34c-26f4226d8724\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"
Jul 24 18:41:29.318605 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.318567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e920df2-78b9-46e7-a34c-26f4226d8724-tmp\") pod \"klusterlet-addon-workmgr-5b88fb6dc8-d594z\" (UID: \"8e920df2-78b9-46e7-a34c-26f4226d8724\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"
Jul 24 18:41:29.321375 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.321352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8e920df2-78b9-46e7-a34c-26f4226d8724-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b88fb6dc8-d594z\" (UID: \"8e920df2-78b9-46e7-a34c-26f4226d8724\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"
Jul 24 18:41:29.337802 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.337776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkmbh\" (UniqueName: \"kubernetes.io/projected/8e920df2-78b9-46e7-a34c-26f4226d8724-kube-api-access-zkmbh\") pod \"klusterlet-addon-workmgr-5b88fb6dc8-d594z\" (UID: \"8e920df2-78b9-46e7-a34c-26f4226d8724\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"
Jul 24 18:41:29.418756 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.418685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d66c79b-9713-4e02-a2d6-82ba72b7f08a-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-kj4n8\" (UID: \"0d66c79b-9713-4e02-a2d6-82ba72b7f08a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kj4n8"
Jul 24 18:41:29.418756 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.418738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-968x2\" (UniqueName: \"kubernetes.io/projected/491c9be2-642b-4b48-828d-e72db595bfe8-kube-api-access-968x2\") pod \"downloads-7c97dc4555-w8s88\" (UID: \"491c9be2-642b-4b48-828d-e72db595bfe8\") " pod="openshift-console/downloads-7c97dc4555-w8s88"
Jul 24 18:41:29.418924 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.418758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d66c79b-9713-4e02-a2d6-82ba72b7f08a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-kj4n8\" (UID: \"0d66c79b-9713-4e02-a2d6-82ba72b7f08a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kj4n8"
Jul 24 18:41:29.472497 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.472461    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"
Jul 24 18:41:29.519278 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.519243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d66c79b-9713-4e02-a2d6-82ba72b7f08a-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-kj4n8\" (UID: \"0d66c79b-9713-4e02-a2d6-82ba72b7f08a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kj4n8"
Jul 24 18:41:29.519442 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.519293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-968x2\" (UniqueName: \"kubernetes.io/projected/491c9be2-642b-4b48-828d-e72db595bfe8-kube-api-access-968x2\") pod \"downloads-7c97dc4555-w8s88\" (UID: \"491c9be2-642b-4b48-828d-e72db595bfe8\") " pod="openshift-console/downloads-7c97dc4555-w8s88"
Jul 24 18:41:29.519442 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.519313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d66c79b-9713-4e02-a2d6-82ba72b7f08a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-kj4n8\" (UID: \"0d66c79b-9713-4e02-a2d6-82ba72b7f08a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kj4n8"
Jul 24 18:41:29.519920 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.519897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d66c79b-9713-4e02-a2d6-82ba72b7f08a-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-kj4n8\" (UID: \"0d66c79b-9713-4e02-a2d6-82ba72b7f08a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kj4n8"
Jul 24 18:41:29.521417 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.521391    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d66c79b-9713-4e02-a2d6-82ba72b7f08a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-kj4n8\" (UID: \"0d66c79b-9713-4e02-a2d6-82ba72b7f08a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kj4n8"
Jul 24 18:41:29.528497 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.528475    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-968x2\" (UniqueName: \"kubernetes.io/projected/491c9be2-642b-4b48-828d-e72db595bfe8-kube-api-access-968x2\") pod \"downloads-7c97dc4555-w8s88\" (UID: \"491c9be2-642b-4b48-828d-e72db595bfe8\") " pod="openshift-console/downloads-7c97dc4555-w8s88"
Jul 24 18:41:29.600831 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.600801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-kj4n8"
Jul 24 18:41:29.607733 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:29.607707    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-w8s88"
Jul 24 18:41:30.293817 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:30.293756    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"]
Jul 24 18:41:30.296595 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:30.296552    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e920df2_78b9_46e7_a34c_26f4226d8724.slice/crio-f59fb663754a7bd93d51c2ebf92945b32d7b536f0caa84d73efe028bb9a8a864 WatchSource:0}: Error finding container f59fb663754a7bd93d51c2ebf92945b32d7b536f0caa84d73efe028bb9a8a864: Status 404 returned error can't find the container with id f59fb663754a7bd93d51c2ebf92945b32d7b536f0caa84d73efe028bb9a8a864
Jul 24 18:41:30.310934 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:30.310913    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-kj4n8"]
Jul 24 18:41:30.313486 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:30.313468    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d66c79b_9713_4e02_a2d6_82ba72b7f08a.slice/crio-449b3060ae008a96b29fd057438a55103cd4d6bb7c2360e8c735fc7553c21b58 WatchSource:0}: Error finding container 449b3060ae008a96b29fd057438a55103cd4d6bb7c2360e8c735fc7553c21b58: Status 404 returned error can't find the container with id 449b3060ae008a96b29fd057438a55103cd4d6bb7c2360e8c735fc7553c21b58
Jul 24 18:41:30.517024 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:30.516945    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-w8s88"]
Jul 24 18:41:30.520564 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:30.520539    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491c9be2_642b_4b48_828d_e72db595bfe8.slice/crio-7b94e96bf16be4c4d37181a1a5541672ed42f3e91a73aab98a5c8b20a00b7895 WatchSource:0}: Error finding container 7b94e96bf16be4c4d37181a1a5541672ed42f3e91a73aab98a5c8b20a00b7895: Status 404 returned error can't find the container with id 7b94e96bf16be4c4d37181a1a5541672ed42f3e91a73aab98a5c8b20a00b7895
Jul 24 18:41:30.533275 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:30.533243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k276f" event={"ID":"04e6b58a-4a62-4635-9058-0899b5799cfe","Type":"ContainerStarted","Data":"620ba3ad47a8d9683f4540d7aad421f3f0e47c5b2dab305c64a6611b45c0fb3d"}
Jul 24 18:41:30.534221 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:30.534195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z" event={"ID":"8e920df2-78b9-46e7-a34c-26f4226d8724","Type":"ContainerStarted","Data":"f59fb663754a7bd93d51c2ebf92945b32d7b536f0caa84d73efe028bb9a8a864"}
Jul 24 18:41:30.535356 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:30.535333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-w8s88" event={"ID":"491c9be2-642b-4b48-828d-e72db595bfe8","Type":"ContainerStarted","Data":"7b94e96bf16be4c4d37181a1a5541672ed42f3e91a73aab98a5c8b20a00b7895"}
Jul 24 18:41:30.536279 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:30.536261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-kj4n8" event={"ID":"0d66c79b-9713-4e02-a2d6-82ba72b7f08a","Type":"ContainerStarted","Data":"449b3060ae008a96b29fd057438a55103cd4d6bb7c2360e8c735fc7553c21b58"}
Jul 24 18:41:32.551044 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:32.550996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-kj4n8" event={"ID":"0d66c79b-9713-4e02-a2d6-82ba72b7f08a","Type":"ContainerStarted","Data":"2725eb76540bbc8994ebaece7ea190bebe6195dcf4b601a50db2026c0fe6a51e"}
Jul 24 18:41:32.566005 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:32.565953    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-kj4n8" podStartSLOduration=2.049053313 podStartE2EDuration="3.56593315s" totalImagesPullingTime="1.516879837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:32.565679763 +0000 UTC m=+91.941339051" watchObservedRunningTime="2026-07-24 18:41:32.56593315 +0000 UTC m=+91.941592448"
Jul 24 18:41:37.575792 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:37.575752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k276f" event={"ID":"04e6b58a-4a62-4635-9058-0899b5799cfe","Type":"ContainerStarted","Data":"ede48ad60bb7fac685f6580ac1dd7761c84d121394df6ae984e467dd84d9869f"}
Jul 24 18:41:37.578654 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:37.578624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z" event={"ID":"8e920df2-78b9-46e7-a34c-26f4226d8724","Type":"ContainerStarted","Data":"414a95a4b06c052fe137e22db7419b1350afaed6a3c033b3b354622af05bdd39"}
Jul 24 18:41:37.578951 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:37.578934    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"
Jul 24 18:41:37.580233 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:37.580211    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z"
Jul 24 18:41:37.593251 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:37.593202    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k276f" podStartSLOduration=17.1368528 podStartE2EDuration="32.593189122s" totalImagesPullingTime="15.456336322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:41:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:37.591915419 +0000 UTC m=+96.967574716" watchObservedRunningTime="2026-07-24 18:41:37.593189122 +0000 UTC m=+96.968848417"
Jul 24 18:41:37.607574 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:37.607536    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b88fb6dc8-d594z" podStartSLOduration=1.483749081 podStartE2EDuration="8.607518356s" totalImagesPullingTime="7.123769275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:37.60648166 +0000 UTC m=+96.982140959" watchObservedRunningTime="2026-07-24 18:41:37.607518356 +0000 UTC m=+96.983177652"
Jul 24 18:41:38.450307 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:38.450266    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f6w5m"
Jul 24 18:41:39.406114 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:39.406074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:41:39.406578 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:39.406121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:41:39.408777 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:39.408752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls\") pod \"image-registry-5f59749bb5-kfnt9\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") " pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:41:39.408892 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:39.408837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44c73487-26c9-4984-a178-1b20cf791453-metrics-tls\") pod \"dns-default-pfdnx\" (UID: \"44c73487-26c9-4984-a178-1b20cf791453\") " pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:41:39.673051 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:39.672959    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tlbzw\""
Jul 24 18:41:39.681630 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:39.681599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:41:39.695476 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:39.695245    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nwwl2\""
Jul 24 18:41:39.703327 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:39.703304    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:41:39.853836 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:39.853709    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f59749bb5-kfnt9"]
Jul 24 18:41:39.855982 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:39.855953    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cd83d8_1c8d_468c_9ea2_8661cf2eecac.slice/crio-d1b337d7699b1b1b8b9370e16a45ee6223e256c2d58f8713d4885e63e6d0fbc4 WatchSource:0}: Error finding container d1b337d7699b1b1b8b9370e16a45ee6223e256c2d58f8713d4885e63e6d0fbc4: Status 404 returned error can't find the container with id d1b337d7699b1b1b8b9370e16a45ee6223e256c2d58f8713d4885e63e6d0fbc4
Jul 24 18:41:39.872275 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:39.872249    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pfdnx"]
Jul 24 18:41:39.875910 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:39.875549    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c73487_26c9_4984_a178_1b20cf791453.slice/crio-4e1726190a13ee374f02c2636dda6efcfcf1761014f6672e22f4790890ac2af2 WatchSource:0}: Error finding container 4e1726190a13ee374f02c2636dda6efcfcf1761014f6672e22f4790890ac2af2: Status 404 returned error can't find the container with id 4e1726190a13ee374f02c2636dda6efcfcf1761014f6672e22f4790890ac2af2
Jul 24 18:41:40.591101 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:40.590932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pfdnx" event={"ID":"44c73487-26c9-4984-a178-1b20cf791453","Type":"ContainerStarted","Data":"4e1726190a13ee374f02c2636dda6efcfcf1761014f6672e22f4790890ac2af2"}
Jul 24 18:41:40.592855 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:40.592819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9" event={"ID":"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac","Type":"ContainerStarted","Data":"4d9a9333a1fb11b4533d1b1483394591e20521930df21f15ca1cac8b5c528d78"}
Jul 24 18:41:40.592980 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:40.592866    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9" event={"ID":"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac","Type":"ContainerStarted","Data":"d1b337d7699b1b1b8b9370e16a45ee6223e256c2d58f8713d4885e63e6d0fbc4"}
Jul 24 18:41:40.592980 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:40.592952    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:41:40.613650 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:40.613598    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9" podStartSLOduration=99.613578547 podStartE2EDuration="1m39.613578547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:40.611597634 +0000 UTC m=+99.987256931" watchObservedRunningTime="2026-07-24 18:41:40.613578547 +0000 UTC m=+99.989237844"
Jul 24 18:41:42.046979 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.046947    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"]
Jul 24 18:41:42.050226 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.050201    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"
Jul 24 18:41:42.052607 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.052582    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 24 18:41:42.053418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.053224    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 18:41:42.053418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.053263    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 18:41:42.053418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.053282    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 24 18:41:42.053418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.053305    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lsdxm\""
Jul 24 18:41:42.053418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.053336    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 18:41:42.060789 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.060756    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"]
Jul 24 18:41:42.128781 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.128752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00540c96-49d5-4d7c-8310-d0a309848478-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-7pcfl\" (UID: \"00540c96-49d5-4d7c-8310-d0a309848478\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"
Jul 24 18:41:42.128946 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.128796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcskr\" (UniqueName: \"kubernetes.io/projected/00540c96-49d5-4d7c-8310-d0a309848478-kube-api-access-rcskr\") pod \"prometheus-operator-84d69f794d-7pcfl\" (UID: \"00540c96-49d5-4d7c-8310-d0a309848478\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"
Jul 24 18:41:42.128946 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.128823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00540c96-49d5-4d7c-8310-d0a309848478-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-7pcfl\" (UID: \"00540c96-49d5-4d7c-8310-d0a309848478\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"
Jul 24 18:41:42.128946 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.128906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/00540c96-49d5-4d7c-8310-d0a309848478-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-7pcfl\" (UID: \"00540c96-49d5-4d7c-8310-d0a309848478\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"
Jul 24 18:41:42.229747 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.229717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00540c96-49d5-4d7c-8310-d0a309848478-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-7pcfl\" (UID: \"00540c96-49d5-4d7c-8310-d0a309848478\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"
Jul 24 18:41:42.229917 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.229760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcskr\" (UniqueName: \"kubernetes.io/projected/00540c96-49d5-4d7c-8310-d0a309848478-kube-api-access-rcskr\") pod \"prometheus-operator-84d69f794d-7pcfl\" (UID: \"00540c96-49d5-4d7c-8310-d0a309848478\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"
Jul 24 18:41:42.229917 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.229831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00540c96-49d5-4d7c-8310-d0a309848478-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-7pcfl\" (UID: \"00540c96-49d5-4d7c-8310-d0a309848478\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"
Jul 24 18:41:42.229917 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.229879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/00540c96-49d5-4d7c-8310-d0a309848478-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-7pcfl\" (UID: \"00540c96-49d5-4d7c-8310-d0a309848478\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"
Jul 24 18:41:42.230719 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.230691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00540c96-49d5-4d7c-8310-d0a309848478-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-7pcfl\" (UID: \"00540c96-49d5-4d7c-8310-d0a309848478\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"
Jul 24 18:41:42.232668 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.232648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/00540c96-49d5-4d7c-8310-d0a309848478-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-7pcfl\" (UID: \"00540c96-49d5-4d7c-8310-d0a309848478\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"
Jul 24 18:41:42.232764 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.232687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00540c96-49d5-4d7c-8310-d0a309848478-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-7pcfl\" (UID: \"00540c96-49d5-4d7c-8310-d0a309848478\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"
Jul 24 18:41:42.242819 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.242799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcskr\" (UniqueName: \"kubernetes.io/projected/00540c96-49d5-4d7c-8310-d0a309848478-kube-api-access-rcskr\") pod \"prometheus-operator-84d69f794d-7pcfl\" (UID: \"00540c96-49d5-4d7c-8310-d0a309848478\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"
Jul 24 18:41:42.361497 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:42.361465    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"
Jul 24 18:41:48.905937 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:48.905910    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-7pcfl"]
Jul 24 18:41:48.908918 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:48.908895    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00540c96_49d5_4d7c_8310_d0a309848478.slice/crio-b12514931540f75dee2d6ffe86e32abac175ff50b10c9442e557f57609e43ade WatchSource:0}: Error finding container b12514931540f75dee2d6ffe86e32abac175ff50b10c9442e557f57609e43ade: Status 404 returned error can't find the container with id b12514931540f75dee2d6ffe86e32abac175ff50b10c9442e557f57609e43ade
Jul 24 18:41:49.631130 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:49.631088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-w8s88" event={"ID":"491c9be2-642b-4b48-828d-e72db595bfe8","Type":"ContainerStarted","Data":"935eff89b3418699f905e029c09f876630764ad52f2d2bf0665fc4aa6204e36e"}
Jul 24 18:41:49.631444 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:49.631412    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-w8s88"
Jul 24 18:41:49.634803 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:49.634771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pfdnx" event={"ID":"44c73487-26c9-4984-a178-1b20cf791453","Type":"ContainerStarted","Data":"1814cf32ee7a7fc806b055a2f1b156f99b05ad9d773cae39272bc30b47fa7832"}
Jul 24 18:41:49.634909 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:49.634826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pfdnx" event={"ID":"44c73487-26c9-4984-a178-1b20cf791453","Type":"ContainerStarted","Data":"e4b122c78c771489b4178634d3ca4f1f9211962a2aa98a3bf95590afe78a9b41"}
Jul 24 18:41:49.635330 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:49.635307    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:41:49.637142 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:49.637122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl" event={"ID":"00540c96-49d5-4d7c-8310-d0a309848478","Type":"ContainerStarted","Data":"b12514931540f75dee2d6ffe86e32abac175ff50b10c9442e557f57609e43ade"}
Jul 24 18:41:49.640754 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:49.640732    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-w8s88"
Jul 24 18:41:49.650645 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:49.650605    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-w8s88" podStartSLOduration=2.356951471 podStartE2EDuration="20.650592793s" totalImagesPullingTime="18.293641322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:49.649396752 +0000 UTC m=+109.025056038" watchObservedRunningTime="2026-07-24 18:41:49.650592793 +0000 UTC m=+109.026252090"
Jul 24 18:41:49.672454 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:49.672405    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pfdnx" podStartSLOduration=65.763142226 podStartE2EDuration="1m14.672369326s" totalImagesPullingTime="8.9092271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:49.671825074 +0000 UTC m=+109.047484371" watchObservedRunningTime="2026-07-24 18:41:49.672369326 +0000 UTC m=+109.048028621"
Jul 24 18:41:51.647593 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:51.647556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl" event={"ID":"00540c96-49d5-4d7c-8310-d0a309848478","Type":"ContainerStarted","Data":"4419ab094f9eb3f81dcca5f61152fa4b6475ea70a425e7b1ac64828fd501702f"}
Jul 24 18:41:51.648036 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:51.647619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl" event={"ID":"00540c96-49d5-4d7c-8310-d0a309848478","Type":"ContainerStarted","Data":"817c6aecf95d2085eb224998a085e6054b437f644eb16e1152ffe81d7dbcde33"}
Jul 24 18:41:51.663567 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:51.663514    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-7pcfl" podStartSLOduration=7.509519503 podStartE2EDuration="9.663495176s" totalImagesPullingTime="2.153975673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:51.66279546 +0000 UTC m=+111.038454758" watchObservedRunningTime="2026-07-24 18:41:51.663495176 +0000 UTC m=+111.039154477"
Jul 24 18:41:53.231990 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.231955    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f59749bb5-kfnt9"]
Jul 24 18:41:53.436279 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.436243    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"]
Jul 24 18:41:53.464415 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.464372    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"]
Jul 24 18:41:53.464552 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.464512    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.464552 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.464534    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6cdbt"]
Jul 24 18:41:53.467081 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.466869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 24 18:41:53.467081 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.466908    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 24 18:41:53.467081 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.466869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9zrv6\""
Jul 24 18:41:53.467081 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.466869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 24 18:41:53.480228 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.480204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.482118 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.482068    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 18:41:53.482118 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.482091    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 18:41:53.482421 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.482403    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 18:41:53.482520 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.482482    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2wdjq\""
Jul 24 18:41:53.523906 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.523882    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.524038 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.523926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.524038 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.523951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v5kq\" (UniqueName: \"kubernetes.io/projected/2cb33685-a791-4ae1-8d34-e2e600caa35b-kube-api-access-9v5kq\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.524038 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.524022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cb33685-a791-4ae1-8d34-e2e600caa35b-metrics-client-ca\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.524213 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.524097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-wtmp\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.524213 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.524127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.524213 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.524152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b11062-82e6-4c0f-a8ab-b78968310aac-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.524350 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.524245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.524350 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.524303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-textfile\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.524350 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.524332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2cb33685-a791-4ae1-8d34-e2e600caa35b-sys\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.524507 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.524357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2cb33685-a791-4ae1-8d34-e2e600caa35b-root\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.524507 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.524413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-accelerators-collector-config\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.524507 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.524442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-tls\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.524507 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.524466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b5b11062-82e6-4c0f-a8ab-b78968310aac-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.524507 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.524490    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vztrz\" (UniqueName: \"kubernetes.io/projected/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-api-access-vztrz\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.624817 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.624787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.624965 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.624837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.624965 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.624871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v5kq\" (UniqueName: \"kubernetes.io/projected/2cb33685-a791-4ae1-8d34-e2e600caa35b-kube-api-access-9v5kq\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.624965 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.624895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cb33685-a791-4ae1-8d34-e2e600caa35b-metrics-client-ca\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.625141 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-wtmp\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.625141 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.625245 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b11062-82e6-4c0f-a8ab-b78968310aac-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.625245 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.625245 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-textfile\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.625418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2cb33685-a791-4ae1-8d34-e2e600caa35b-sys\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.625418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-wtmp\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.625418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2cb33685-a791-4ae1-8d34-e2e600caa35b-root\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.625418 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:53.625292    2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 24 18:41:53.625418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-accelerators-collector-config\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.625418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-tls\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.625418 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:53.625354    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-state-metrics-tls podName:b5b11062-82e6-4c0f-a8ab-b78968310aac nodeName:}" failed. No retries permitted until 2026-07-24 18:41:54.125335157 +0000 UTC m=+113.500994452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-x8dd2" (UID: "b5b11062-82e6-4c0f-a8ab-b78968310aac") : secret "kube-state-metrics-tls" not found
Jul 24 18:41:53.625418 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b5b11062-82e6-4c0f-a8ab-b78968310aac-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.625825 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:53.625432    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 24 18:41:53.625825 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vztrz\" (UniqueName: \"kubernetes.io/projected/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-api-access-vztrz\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.625825 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:41:53.625480    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-tls podName:2cb33685-a791-4ae1-8d34-e2e600caa35b nodeName:}" failed. No retries permitted until 2026-07-24 18:41:54.125464182 +0000 UTC m=+113.501123475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-tls") pod "node-exporter-6cdbt" (UID: "2cb33685-a791-4ae1-8d34-e2e600caa35b") : secret "node-exporter-tls" not found
Jul 24 18:41:53.625825 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625530    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2cb33685-a791-4ae1-8d34-e2e600caa35b-root\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.625825 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-textfile\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.625825 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cb33685-a791-4ae1-8d34-e2e600caa35b-metrics-client-ca\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.625825 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2cb33685-a791-4ae1-8d34-e2e600caa35b-sys\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.625825 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.626265 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b11062-82e6-4c0f-a8ab-b78968310aac-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.626265 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b5b11062-82e6-4c0f-a8ab-b78968310aac-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.626265 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.625992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-accelerators-collector-config\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.628144 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.628100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.628239 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.628163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:53.632934 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.632905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v5kq\" (UniqueName: \"kubernetes.io/projected/2cb33685-a791-4ae1-8d34-e2e600caa35b-kube-api-access-9v5kq\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:53.633849 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:53.633829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vztrz\" (UniqueName: \"kubernetes.io/projected/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-api-access-vztrz\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:54.129540 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:54.129495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:54.129748 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:54.129586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-tls\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:54.132281 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:54.132253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2cb33685-a791-4ae1-8d34-e2e600caa35b-node-exporter-tls\") pod \"node-exporter-6cdbt\" (UID: \"2cb33685-a791-4ae1-8d34-e2e600caa35b\") " pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:54.132442 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:54.132423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b11062-82e6-4c0f-a8ab-b78968310aac-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-x8dd2\" (UID: \"b5b11062-82e6-4c0f-a8ab-b78968310aac\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:54.375816 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:54.375782    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"
Jul 24 18:41:54.391062 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:54.391038    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6cdbt"
Jul 24 18:41:54.415319 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:54.415256    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb33685_a791_4ae1_8d34_e2e600caa35b.slice/crio-edec63961cb5737025da614f0eebc401754a40ed04d98a4d2c5d4823e3c1fb5c WatchSource:0}: Error finding container edec63961cb5737025da614f0eebc401754a40ed04d98a4d2c5d4823e3c1fb5c: Status 404 returned error can't find the container with id edec63961cb5737025da614f0eebc401754a40ed04d98a4d2c5d4823e3c1fb5c
Jul 24 18:41:54.543024 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:54.542981    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2"]
Jul 24 18:41:54.547066 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:54.547022    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b11062_82e6_4c0f_a8ab_b78968310aac.slice/crio-f3947f727c822d1685812c8b370c0523d1c89ea0258ed8e958b7ce4c1efad9a0 WatchSource:0}: Error finding container f3947f727c822d1685812c8b370c0523d1c89ea0258ed8e958b7ce4c1efad9a0: Status 404 returned error can't find the container with id f3947f727c822d1685812c8b370c0523d1c89ea0258ed8e958b7ce4c1efad9a0
Jul 24 18:41:54.661911 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:54.661822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2" event={"ID":"b5b11062-82e6-4c0f-a8ab-b78968310aac","Type":"ContainerStarted","Data":"f3947f727c822d1685812c8b370c0523d1c89ea0258ed8e958b7ce4c1efad9a0"}
Jul 24 18:41:54.663193 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:54.663163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6cdbt" event={"ID":"2cb33685-a791-4ae1-8d34-e2e600caa35b","Type":"ContainerStarted","Data":"edec63961cb5737025da614f0eebc401754a40ed04d98a4d2c5d4823e3c1fb5c"}
Jul 24 18:41:56.673174 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:56.673141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6cdbt" event={"ID":"2cb33685-a791-4ae1-8d34-e2e600caa35b","Type":"ContainerStarted","Data":"52329e49089de30a1dceaf689f8e7279bfd07105a9abdc06c94c4b71c64994a1"}
Jul 24 18:41:57.679989 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:57.679950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2" event={"ID":"b5b11062-82e6-4c0f-a8ab-b78968310aac","Type":"ContainerStarted","Data":"46d1d9ba9b5e027bc197fd968224245d63466cbaf0a1f3f4615dc8509b3cc455"}
Jul 24 18:41:57.680570 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:57.680025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2" event={"ID":"b5b11062-82e6-4c0f-a8ab-b78968310aac","Type":"ContainerStarted","Data":"83955e63183e98c0bdd3e8480f2b95cc117288851becd1a2bad0ca27cb2dcc1a"}
Jul 24 18:41:57.680570 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:57.680059    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2" event={"ID":"b5b11062-82e6-4c0f-a8ab-b78968310aac","Type":"ContainerStarted","Data":"54806af83786c3f6068215af5a46c774d6baf03efbc1bc8ad7b9a248cac3e035"}
Jul 24 18:41:57.681781 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:57.681751    2580 generic.go:356] "Generic (PLEG): container finished" podID="2cb33685-a791-4ae1-8d34-e2e600caa35b" containerID="52329e49089de30a1dceaf689f8e7279bfd07105a9abdc06c94c4b71c64994a1" exitCode=0
Jul 24 18:41:57.681898 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:57.681785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6cdbt" event={"ID":"2cb33685-a791-4ae1-8d34-e2e600caa35b","Type":"ContainerDied","Data":"52329e49089de30a1dceaf689f8e7279bfd07105a9abdc06c94c4b71c64994a1"}
Jul 24 18:41:57.698781 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:57.698734    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x8dd2" podStartSLOduration=2.471736196 podStartE2EDuration="4.698719134s" totalImagesPullingTime="2.226982938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:57.698685592 +0000 UTC m=+117.074344889" watchObservedRunningTime="2026-07-24 18:41:57.698719134 +0000 UTC m=+117.074378430"
Jul 24 18:41:58.639984 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.639953    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-565765d559-zdkfg"]
Jul 24 18:41:58.679867 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.679841    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-565765d559-zdkfg"]
Jul 24 18:41:58.680021 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.679953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.682695 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.682674    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 24 18:41:58.682987 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.682960    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 24 18:41:58.683085 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.682965    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 24 18:41:58.683085 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.683025    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 24 18:41:58.683085 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.682964    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bq442\""
Jul 24 18:41:58.683437 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.683416    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 24 18:41:58.688587 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.688560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6cdbt" event={"ID":"2cb33685-a791-4ae1-8d34-e2e600caa35b","Type":"ContainerStarted","Data":"0deff8f97921d1b5789e94b87abd147a45e9f1dd3d8750993de380c2ce0926d2"}
Jul 24 18:41:58.688675 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.688625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6cdbt" event={"ID":"2cb33685-a791-4ae1-8d34-e2e600caa35b","Type":"ContainerStarted","Data":"34e8c33c20d96951ea9a93ae314f18603034475a79dfabc5db4312b65bbcdb17"}
Jul 24 18:41:58.688757 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.688738    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 24 18:41:58.718641 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.718597    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6cdbt" podStartSLOduration=3.600122942 podStartE2EDuration="5.718581848s" totalImagesPullingTime="2.011281455s" totalInitContainerRuntime="107.177451ms" isStatefulPod=true podCreationTimestamp="2026-07-24 18:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:41:58.717845808 +0000 UTC m=+118.093505103" watchObservedRunningTime="2026-07-24 18:41:58.718581848 +0000 UTC m=+118.094241145"
Jul 24 18:41:58.771208 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.771179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.771304 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.771248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.771551 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.771532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-secret-telemeter-client\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.771621 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.771574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-federate-client-tls\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.771675 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.771632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqkrf\" (UniqueName: \"kubernetes.io/projected/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-kube-api-access-nqkrf\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.771724 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.771692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-telemeter-client-tls\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.771802 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.771778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-serving-certs-ca-bundle\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.771862 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.771819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-metrics-client-ca\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.872503 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.872472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-federate-client-tls\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.872619 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.872508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqkrf\" (UniqueName: \"kubernetes.io/projected/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-kube-api-access-nqkrf\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.872619 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.872530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-telemeter-client-tls\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.872619 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.872549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-serving-certs-ca-bundle\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.872619 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.872573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-metrics-client-ca\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.872619 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.872612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.872918 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.872653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.872918 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.872723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-secret-telemeter-client\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.873358 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.873333    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-serving-certs-ca-bundle\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.873472 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.873354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-metrics-client-ca\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.873681 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.873656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.875006 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.874983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-federate-client-tls\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.875200 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.875173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-secret-telemeter-client\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.875449 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.875427    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.875498 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.875432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-telemeter-client-tls\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.884343 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.884325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqkrf\" (UniqueName: \"kubernetes.io/projected/5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9-kube-api-access-nqkrf\") pod \"telemeter-client-565765d559-zdkfg\" (UID: \"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9\") " pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:58.990272 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:58.990222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-565765d559-zdkfg"
Jul 24 18:41:59.138588 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.138553    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-565765d559-zdkfg"]
Jul 24 18:41:59.141550 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:41:59.141522    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5071a7b0_f0b6_4cf9_9e3e_5d235703a6b9.slice/crio-3e89ca784487b232f389e8e66baaabbfe8c2d6584cc2d0cfa2d1d8fff7199637 WatchSource:0}: Error finding container 3e89ca784487b232f389e8e66baaabbfe8c2d6584cc2d0cfa2d1d8fff7199637: Status 404 returned error can't find the container with id 3e89ca784487b232f389e8e66baaabbfe8c2d6584cc2d0cfa2d1d8fff7199637
Jul 24 18:41:59.649066 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.649024    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 18:41:59.673989 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.673954    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 18:41:59.674133 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.674100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.678486 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.677825    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 18:41:59.678486 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.677887    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 18:41:59.678486 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.677969    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 18:41:59.678486 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.678037    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 18:41:59.678486 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.678098    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 18:41:59.678486 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.678195    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 18:41:59.678486 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.678201    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 18:41:59.678486 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.678303    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 18:41:59.678486 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.678365    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 18:41:59.678486 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.678455    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zzqbg\""
Jul 24 18:41:59.679133 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.678584    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 24 18:41:59.679133 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.678660    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fsj8uq6brab48\""
Jul 24 18:41:59.679133 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.678829    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 18:41:59.679133 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.678830    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 18:41:59.679324 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.679278    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 18:41:59.681513 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.681482    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 18:41:59.683591 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.683568    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 18:41:59.696318 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.696291    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-565765d559-zdkfg" event={"ID":"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9","Type":"ContainerStarted","Data":"3e89ca784487b232f389e8e66baaabbfe8c2d6584cc2d0cfa2d1d8fff7199637"}
Jul 24 18:41:59.779073 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43cbe10e-ddce-4634-9a4c-8264ace66023-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.779073 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-web-config\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.779333 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.779333 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.779455 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.779455 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.779455 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42x7p\" (UniqueName: \"kubernetes.io/projected/43cbe10e-ddce-4634-9a4c-8264ace66023-kube-api-access-42x7p\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.779584 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.779584 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.779663 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779599    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-config\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.779718 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.779789 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.779885 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.780011 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.780011 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.779977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.780141 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.780010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.780141 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.780034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.780141 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.780104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43cbe10e-ddce-4634-9a4c-8264ace66023-config-out\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.780141 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.780133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.780315 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.780159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881295 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43cbe10e-ddce-4634-9a4c-8264ace66023-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881512 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-web-config\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881512 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881512 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881512 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881512 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881512 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42x7p\" (UniqueName: \"kubernetes.io/projected/43cbe10e-ddce-4634-9a4c-8264ace66023-kube-api-access-42x7p\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881843 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881843 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881843 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881604    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-config\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881843 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881843 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881843 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881843 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.881843 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.882251 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.882251 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.882251 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43cbe10e-ddce-4634-9a4c-8264ace66023-config-out\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.882251 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.882251 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.881966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.882850 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.882776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.884135 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.883821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.886410 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.886004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43cbe10e-ddce-4634-9a4c-8264ace66023-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.887644 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.886728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.887644 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.886756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.887644 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.887312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.888991 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.888964    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.889423 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.889399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.889770 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.889745    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.890337 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.890312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.890753 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.890440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.890753 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.890718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.890753 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.890740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-config\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.890949 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.890839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43cbe10e-ddce-4634-9a4c-8264ace66023-config-out\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.891750 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.891507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43cbe10e-ddce-4634-9a4c-8264ace66023-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.892028 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.891978    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.892028 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.892019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42x7p\" (UniqueName: \"kubernetes.io/projected/43cbe10e-ddce-4634-9a4c-8264ace66023-kube-api-access-42x7p\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.894044 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.893639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-web-config\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.894044 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.893639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.894701 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.894675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43cbe10e-ddce-4634-9a4c-8264ace66023-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43cbe10e-ddce-4634-9a4c-8264ace66023\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:41:59.988876 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:41:59.988613    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:42:00.174967 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:00.174930    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 18:42:00.179105 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:42:00.179072    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cbe10e_ddce_4634_9a4c_8264ace66023.slice/crio-fcc8eda288dbd3f399bb0029f9846507fe1f7deef05ded521cd7d073b5236c8d WatchSource:0}: Error finding container fcc8eda288dbd3f399bb0029f9846507fe1f7deef05ded521cd7d073b5236c8d: Status 404 returned error can't find the container with id fcc8eda288dbd3f399bb0029f9846507fe1f7deef05ded521cd7d073b5236c8d
Jul 24 18:42:00.650056 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:00.650026    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pfdnx"
Jul 24 18:42:00.702654 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:00.702560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43cbe10e-ddce-4634-9a4c-8264ace66023","Type":"ContainerStarted","Data":"fcc8eda288dbd3f399bb0029f9846507fe1f7deef05ded521cd7d073b5236c8d"}
Jul 24 18:42:02.710746 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:02.710715    2580 generic.go:356] "Generic (PLEG): container finished" podID="43cbe10e-ddce-4634-9a4c-8264ace66023" containerID="4b56a357cf56d294d375f981dd286971e52583532312f5b0b496b492d2acb069" exitCode=0
Jul 24 18:42:02.711264 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:02.710781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43cbe10e-ddce-4634-9a4c-8264ace66023","Type":"ContainerDied","Data":"4b56a357cf56d294d375f981dd286971e52583532312f5b0b496b492d2acb069"}
Jul 24 18:42:02.713075 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:02.713053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-565765d559-zdkfg" event={"ID":"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9","Type":"ContainerStarted","Data":"51b9817af9aece865ef273cef8541963198d4ca4b53417e7aacc7baddd3ab583"}
Jul 24 18:42:02.713175 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:02.713103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-565765d559-zdkfg" event={"ID":"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9","Type":"ContainerStarted","Data":"3fce98612f33fc129b86fd35054516873f043852d86154d6586cb1523806f0dc"}
Jul 24 18:42:02.713175 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:02.713141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-565765d559-zdkfg" event={"ID":"5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9","Type":"ContainerStarted","Data":"3f6db4f9ddf9f4e8d100ff04b542f4f24c689dcce24a072f8ff34b4fc1679dc4"}
Jul 24 18:42:02.757859 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:02.757816    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-565765d559-zdkfg" podStartSLOduration=1.958997755 podStartE2EDuration="4.757800876s" totalImagesPullingTime="2.798803121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:42:02.75648814 +0000 UTC m=+122.132147436" watchObservedRunningTime="2026-07-24 18:42:02.757800876 +0000 UTC m=+122.133460171"
Jul 24 18:42:03.237494 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:03.237464    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:42:06.732818 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:06.732786    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43cbe10e-ddce-4634-9a4c-8264ace66023","Type":"ContainerStarted","Data":"7f1949cc0012fd7565e658b404b5b902738d3907bdd4eb6f5b18786270ce9b00"}
Jul 24 18:42:06.733235 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:06.732854    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43cbe10e-ddce-4634-9a4c-8264ace66023","Type":"ContainerStarted","Data":"65aa1081e77ee0fd082a5f777a0062067732f631c710faa40671ccc6d2421e10"}
Jul 24 18:42:09.746982 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:09.746941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43cbe10e-ddce-4634-9a4c-8264ace66023","Type":"ContainerStarted","Data":"93125a08aa2a887efeb84d4f21adee04f334d9c05a162098f980ecd7bbf216e2"}
Jul 24 18:42:09.747365 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:09.747006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43cbe10e-ddce-4634-9a4c-8264ace66023","Type":"ContainerStarted","Data":"07e055b46f52bc786c1905b59e05f044819a2be68f6476b948fb114a7a641c69"}
Jul 24 18:42:09.747365 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:09.747032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43cbe10e-ddce-4634-9a4c-8264ace66023","Type":"ContainerStarted","Data":"3f6b7f11c7e0b11143a3a885170347b8888a65b8be7352afb813c064f849b9cb"}
Jul 24 18:42:09.747365 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:09.747085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43cbe10e-ddce-4634-9a4c-8264ace66023","Type":"ContainerStarted","Data":"2292014203d10b8442f1b9785af1a1e53a41085ec3c084ff5ce52a42ded66bc6"}
Jul 24 18:42:09.777276 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:09.777193    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.018719843 podStartE2EDuration="10.777177556s" totalImagesPullingTime="7.741744536s" totalInitContainerRuntime="16.713177ms" isStatefulPod=false podCreationTimestamp="2026-07-24 18:41:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:42:09.775327358 +0000 UTC m=+129.150986653" watchObservedRunningTime="2026-07-24 18:42:09.777177556 +0000 UTC m=+129.152836898"
Jul 24 18:42:09.988861 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:09.988828    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:42:10.883963 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:10.883923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs\") pod \"network-metrics-daemon-xkxwb\" (UID: \"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0\") " pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:42:10.886609 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:10.886581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0785e8-c533-44cb-bc6f-d11ed1bb60f0-metrics-certs\") pod \"network-metrics-daemon-xkxwb\" (UID: \"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0\") " pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:42:11.011303 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:11.011273    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f8czh\""
Jul 24 18:42:11.019145 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:11.019125    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkxwb"
Jul 24 18:42:11.168175 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:11.168101    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xkxwb"]
Jul 24 18:42:11.170644 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:42:11.170622    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0785e8_c533_44cb_bc6f_d11ed1bb60f0.slice/crio-e8c6b404995c3d633b4f804dacefd37683d1bd0f40651f5db6015a209dd7c865 WatchSource:0}: Error finding container e8c6b404995c3d633b4f804dacefd37683d1bd0f40651f5db6015a209dd7c865: Status 404 returned error can't find the container with id e8c6b404995c3d633b4f804dacefd37683d1bd0f40651f5db6015a209dd7c865
Jul 24 18:42:11.755919 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:11.755887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkxwb" event={"ID":"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0","Type":"ContainerStarted","Data":"e8c6b404995c3d633b4f804dacefd37683d1bd0f40651f5db6015a209dd7c865"}
Jul 24 18:42:12.762060 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:12.762032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkxwb" event={"ID":"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0","Type":"ContainerStarted","Data":"5857a64f086285f3d3af0031cbd5895721db8ef227151ee27b8ffaf54745601c"}
Jul 24 18:42:12.762396 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:12.762109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkxwb" event={"ID":"2e0785e8-c533-44cb-bc6f-d11ed1bb60f0","Type":"ContainerStarted","Data":"2f5401e974b391485f462532a5fa49582b51bca34b5ba61919e52c6d5b7419ff"}
Jul 24 18:42:13.782738 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:13.782687    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xkxwb" podStartSLOduration=131.478566936 podStartE2EDuration="2m12.782670978s" totalImagesPullingTime="1.304104042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:42:13.781708359 +0000 UTC m=+133.157367655" watchObservedRunningTime="2026-07-24 18:42:13.782670978 +0000 UTC m=+133.158330472"
Jul 24 18:42:18.253881 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.253822    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9" podUID="c1cd83d8-1c8d-468c-9ea2-8661cf2eecac" containerName="registry" containerID="cri-o://4d9a9333a1fb11b4533d1b1483394591e20521930df21f15ca1cac8b5c528d78" gracePeriod=30
Jul 24 18:42:18.522216 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.522164    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:42:18.654260 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.654231    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-installation-pull-secrets\") pod \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") "
Jul 24 18:42:18.654454 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.654269    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-ca-trust-extracted\") pod \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") "
Jul 24 18:42:18.654454 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.654318    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t277\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-kube-api-access-2t277\") pod \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") "
Jul 24 18:42:18.654454 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.654335    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-bound-sa-token\") pod \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") "
Jul 24 18:42:18.654454 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.654360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-trusted-ca\") pod \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") "
Jul 24 18:42:18.654454 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.654394    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-image-registry-private-configuration\") pod \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") "
Jul 24 18:42:18.654717 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.654519    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls\") pod \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") "
Jul 24 18:42:18.654717 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.654563    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-certificates\") pod \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\" (UID: \"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac\") "
Jul 24 18:42:18.654867 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.654828    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:42:18.655194 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.655164    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:42:18.656908 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.656879    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:42:18.656908 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.656895    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:42:18.657054 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.657016    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-kube-api-access-2t277" (OuterVolumeSpecName: "kube-api-access-2t277") pod "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac"). InnerVolumeSpecName "kube-api-access-2t277". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:42:18.657114 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.657094    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:42:18.657150 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.657097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:42:18.665113 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.665087    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac" (UID: "c1cd83d8-1c8d-468c-9ea2-8661cf2eecac"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:42:18.755176 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.755149    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t277\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-kube-api-access-2t277\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:42:18.755176 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.755173    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-bound-sa-token\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:42:18.755316 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.755183    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-trusted-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:42:18.755316 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.755192    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-image-registry-private-configuration\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:42:18.755316 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.755201    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:42:18.755316 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.755209    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-registry-certificates\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:42:18.755316 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.755218    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-installation-pull-secrets\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:42:18.755316 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.755226    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac-ca-trust-extracted\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:42:18.787234 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.787173    2580 generic.go:356] "Generic (PLEG): container finished" podID="c1cd83d8-1c8d-468c-9ea2-8661cf2eecac" containerID="4d9a9333a1fb11b4533d1b1483394591e20521930df21f15ca1cac8b5c528d78" exitCode=0
Jul 24 18:42:18.787234 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.787213    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9"
Jul 24 18:42:18.787349 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.787234    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9" event={"ID":"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac","Type":"ContainerDied","Data":"4d9a9333a1fb11b4533d1b1483394591e20521930df21f15ca1cac8b5c528d78"}
Jul 24 18:42:18.787349 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.787279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f59749bb5-kfnt9" event={"ID":"c1cd83d8-1c8d-468c-9ea2-8661cf2eecac","Type":"ContainerDied","Data":"d1b337d7699b1b1b8b9370e16a45ee6223e256c2d58f8713d4885e63e6d0fbc4"}
Jul 24 18:42:18.787349 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.787300    2580 scope.go:117] "RemoveContainer" containerID="4d9a9333a1fb11b4533d1b1483394591e20521930df21f15ca1cac8b5c528d78"
Jul 24 18:42:18.795706 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.795690    2580 scope.go:117] "RemoveContainer" containerID="4d9a9333a1fb11b4533d1b1483394591e20521930df21f15ca1cac8b5c528d78"
Jul 24 18:42:18.795978 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:42:18.795950    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d9a9333a1fb11b4533d1b1483394591e20521930df21f15ca1cac8b5c528d78\": container with ID starting with 4d9a9333a1fb11b4533d1b1483394591e20521930df21f15ca1cac8b5c528d78 not found: ID does not exist" containerID="4d9a9333a1fb11b4533d1b1483394591e20521930df21f15ca1cac8b5c528d78"
Jul 24 18:42:18.796026 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.795987    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9a9333a1fb11b4533d1b1483394591e20521930df21f15ca1cac8b5c528d78"} err="failed to get container status \"4d9a9333a1fb11b4533d1b1483394591e20521930df21f15ca1cac8b5c528d78\": rpc error: code = NotFound desc = could not find container \"4d9a9333a1fb11b4533d1b1483394591e20521930df21f15ca1cac8b5c528d78\": container with ID starting with 4d9a9333a1fb11b4533d1b1483394591e20521930df21f15ca1cac8b5c528d78 not found: ID does not exist"
Jul 24 18:42:18.807711 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.807688    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f59749bb5-kfnt9"]
Jul 24 18:42:18.811554 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:18.811526    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f59749bb5-kfnt9"]
Jul 24 18:42:19.196297 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:19.196264    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1cd83d8-1c8d-468c-9ea2-8661cf2eecac" path="/var/lib/kubelet/pods/c1cd83d8-1c8d-468c-9ea2-8661cf2eecac/volumes"
Jul 24 18:42:30.835535 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:30.835491    2580 generic.go:356] "Generic (PLEG): container finished" podID="7fd19688-7aa4-4567-8ada-87afa3c0a7a9" containerID="0c1825b964188592e69e97e0c7455cb4920e859b5a25019a268dee980bd11cfc" exitCode=0
Jul 24 18:42:30.835935 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:30.835563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts" event={"ID":"7fd19688-7aa4-4567-8ada-87afa3c0a7a9","Type":"ContainerDied","Data":"0c1825b964188592e69e97e0c7455cb4920e859b5a25019a268dee980bd11cfc"}
Jul 24 18:42:30.835935 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:30.835880    2580 scope.go:117] "RemoveContainer" containerID="0c1825b964188592e69e97e0c7455cb4920e859b5a25019a268dee980bd11cfc"
Jul 24 18:42:31.841267 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:31.841236    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts" event={"ID":"7fd19688-7aa4-4567-8ada-87afa3c0a7a9","Type":"ContainerStarted","Data":"154de3b7bf70fdc31e3f623f50cf602618ad43ea95581a3905c7d934bbbb6431"}
Jul 24 18:42:59.989441 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:42:59.989352    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:43:00.008145 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:43:00.008122    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:43:00.977558 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:43:00.977530    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 18:44:54.017295 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.017258    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-282wj"]
Jul 24 18:44:54.017759 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.017621    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1cd83d8-1c8d-468c-9ea2-8661cf2eecac" containerName="registry"
Jul 24 18:44:54.017759 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.017635    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1cd83d8-1c8d-468c-9ea2-8661cf2eecac" containerName="registry"
Jul 24 18:44:54.017759 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.017692    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1cd83d8-1c8d-468c-9ea2-8661cf2eecac" containerName="registry"
Jul 24 18:44:54.020357 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.020340    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-282wj"
Jul 24 18:44:54.022757 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.022736    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 24 18:44:54.022995 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.022979    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-dwgdc\""
Jul 24 18:44:54.022995 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.022987    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 24 18:44:54.030765 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.030743    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-282wj"]
Jul 24 18:44:54.107700 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.107667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/554f313d-854f-4821-803c-63c6c0fe081f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-282wj\" (UID: \"554f313d-854f-4821-803c-63c6c0fe081f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-282wj"
Jul 24 18:44:54.107869 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.107705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq7hn\" (UniqueName: \"kubernetes.io/projected/554f313d-854f-4821-803c-63c6c0fe081f-kube-api-access-hq7hn\") pod \"openshift-lws-operator-fd8ccff4c-282wj\" (UID: \"554f313d-854f-4821-803c-63c6c0fe081f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-282wj"
Jul 24 18:44:54.208211 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.208177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/554f313d-854f-4821-803c-63c6c0fe081f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-282wj\" (UID: \"554f313d-854f-4821-803c-63c6c0fe081f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-282wj"
Jul 24 18:44:54.208211 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.208211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq7hn\" (UniqueName: \"kubernetes.io/projected/554f313d-854f-4821-803c-63c6c0fe081f-kube-api-access-hq7hn\") pod \"openshift-lws-operator-fd8ccff4c-282wj\" (UID: \"554f313d-854f-4821-803c-63c6c0fe081f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-282wj"
Jul 24 18:44:54.208557 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.208536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/554f313d-854f-4821-803c-63c6c0fe081f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-282wj\" (UID: \"554f313d-854f-4821-803c-63c6c0fe081f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-282wj"
Jul 24 18:44:54.219759 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.219732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq7hn\" (UniqueName: \"kubernetes.io/projected/554f313d-854f-4821-803c-63c6c0fe081f-kube-api-access-hq7hn\") pod \"openshift-lws-operator-fd8ccff4c-282wj\" (UID: \"554f313d-854f-4821-803c-63c6c0fe081f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-282wj"
Jul 24 18:44:54.337609 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.337589    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-282wj"
Jul 24 18:44:54.463064 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:54.463033    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-282wj"]
Jul 24 18:44:54.466169 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:44:54.466142    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod554f313d_854f_4821_803c_63c6c0fe081f.slice/crio-195a7e9e5fcf47475e1c0e26b1bb7f7f2980b8cdcdf2da58dceb12232c928d92 WatchSource:0}: Error finding container 195a7e9e5fcf47475e1c0e26b1bb7f7f2980b8cdcdf2da58dceb12232c928d92: Status 404 returned error can't find the container with id 195a7e9e5fcf47475e1c0e26b1bb7f7f2980b8cdcdf2da58dceb12232c928d92
Jul 24 18:44:55.375896 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:55.375853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-282wj" event={"ID":"554f313d-854f-4821-803c-63c6c0fe081f","Type":"ContainerStarted","Data":"195a7e9e5fcf47475e1c0e26b1bb7f7f2980b8cdcdf2da58dceb12232c928d92"}
Jul 24 18:44:57.385610 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:57.385576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-282wj" event={"ID":"554f313d-854f-4821-803c-63c6c0fe081f","Type":"ContainerStarted","Data":"d25f0a18f556c09bb1779961d0f27666a35602f68276f9d26c56f540bc02ca94"}
Jul 24 18:44:57.404639 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:44:57.404580    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-282wj" podStartSLOduration=2.012125296 podStartE2EDuration="4.404561476s" totalImagesPullingTime="2.39243618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:57.403961728 +0000 UTC m=+296.779621023" watchObservedRunningTime="2026-07-24 18:44:57.404561476 +0000 UTC m=+296.780220776"
Jul 24 18:45:01.039252 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:01.039224    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 18:45:01.049287 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:01.049250    2580 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 18:45:01.049723 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:01.049707    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 18:45:14.763283 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.763255    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"]
Jul 24 18:45:14.769781 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.769765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.777323 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.777301    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 18:45:14.777323 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.777314    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 18:45:14.778415 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.778400    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 18:45:14.778636 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.778617    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jb8gh\""
Jul 24 18:45:14.780906 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.780891    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 18:45:14.791222 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.791200    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"]
Jul 24 18:45:14.862515 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.862484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-tmp\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.862666 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.862522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.862666 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.862547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-manifests\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.862666 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.862566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.862666 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.862620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5chwj\" (UniqueName: \"kubernetes.io/projected/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-kube-api-access-5chwj\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.963543 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.963481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-tmp\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.963739 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.963557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.963739 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.963603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-manifests\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.963739 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.963634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.963739 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.963659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5chwj\" (UniqueName: \"kubernetes.io/projected/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-kube-api-access-5chwj\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.964613 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.964268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-manifests\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.966778 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.966757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-tmp\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.968563 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.968540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.968672 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.968608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:14.982513 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:14.982478    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5chwj\" (UniqueName: \"kubernetes.io/projected/e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd-kube-api-access-5chwj\") pod \"opendatahub-operator-controller-manager-859dddff6f-km4rp\" (UID: \"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd\") " pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:15.080140 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:15.080106    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:15.224779 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:15.224756    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"]
Jul 24 18:45:15.227071 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:45:15.227040    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b4b3d4_3ce1_48fd_98e2_e6b8383354cd.slice/crio-738f974e642aeab6a70138ed9180c27c9796607742d1e3425c8f246cadd00f6f WatchSource:0}: Error finding container 738f974e642aeab6a70138ed9180c27c9796607742d1e3425c8f246cadd00f6f: Status 404 returned error can't find the container with id 738f974e642aeab6a70138ed9180c27c9796607742d1e3425c8f246cadd00f6f
Jul 24 18:45:15.228852 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:15.228833    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:45:15.454430 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:15.454351    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp" event={"ID":"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd","Type":"ContainerStarted","Data":"738f974e642aeab6a70138ed9180c27c9796607742d1e3425c8f246cadd00f6f"}
Jul 24 18:45:19.473769 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:19.473730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp" event={"ID":"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd","Type":"ContainerStarted","Data":"f79b79bd001f5d6849f4734960110ae7d4996037382f72e226af35b73434ebf4"}
Jul 24 18:45:20.479513 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:20.479480    2580 generic.go:356] "Generic (PLEG): container finished" podID="e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd" containerID="f79b79bd001f5d6849f4734960110ae7d4996037382f72e226af35b73434ebf4" exitCode=0
Jul 24 18:45:20.479886 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:20.479519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp" event={"ID":"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd","Type":"ContainerDied","Data":"f79b79bd001f5d6849f4734960110ae7d4996037382f72e226af35b73434ebf4"}
Jul 24 18:45:21.485258 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:21.485218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp" event={"ID":"e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd","Type":"ContainerStarted","Data":"80d93454dfbf6acf1c82041808cd4ea62ca2c0c122e47c180c6f71312f8ae9d4"}
Jul 24 18:45:21.485665 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:21.485494    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:21.512028 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:21.511988    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp" podStartSLOduration=2.77675883 podStartE2EDuration="7.511974686s" totalImagesPullingTime="3.643880373s" totalInitContainerRuntime="1.091335483s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:45:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:45:21.510568484 +0000 UTC m=+320.886227781" watchObservedRunningTime="2026-07-24 18:45:21.511974686 +0000 UTC m=+320.887633981"
Jul 24 18:45:32.491678 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:32.491648    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-859dddff6f-km4rp"
Jul 24 18:45:32.972952 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:32.972923    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"]
Jul 24 18:45:32.981183 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:32.981157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:32.983408 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:32.983364    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 24 18:45:32.983661 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:32.983641    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 24 18:45:32.983661 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:32.983658    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 24 18:45:32.983771 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:32.983647    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zs4t2\""
Jul 24 18:45:32.986523 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:32.986504    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"]
Jul 24 18:45:33.106114 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.106089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72jfd\" (UniqueName: \"kubernetes.io/projected/a2ce7387-1f69-4770-8939-902c30a71e61-kube-api-access-72jfd\") pod \"lws-controller-manager-8456f466bb-25cmx\" (UID: \"a2ce7387-1f69-4770-8939-902c30a71e61\") " pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:33.106259 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.106119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2ce7387-1f69-4770-8939-902c30a71e61-manager-config\") pod \"lws-controller-manager-8456f466bb-25cmx\" (UID: \"a2ce7387-1f69-4770-8939-902c30a71e61\") " pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:33.106259 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.106144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2ce7387-1f69-4770-8939-902c30a71e61-cert\") pod \"lws-controller-manager-8456f466bb-25cmx\" (UID: \"a2ce7387-1f69-4770-8939-902c30a71e61\") " pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:33.106259 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.106240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2ce7387-1f69-4770-8939-902c30a71e61-metrics-cert\") pod \"lws-controller-manager-8456f466bb-25cmx\" (UID: \"a2ce7387-1f69-4770-8939-902c30a71e61\") " pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:33.207342 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.207315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72jfd\" (UniqueName: \"kubernetes.io/projected/a2ce7387-1f69-4770-8939-902c30a71e61-kube-api-access-72jfd\") pod \"lws-controller-manager-8456f466bb-25cmx\" (UID: \"a2ce7387-1f69-4770-8939-902c30a71e61\") " pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:33.207508 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.207346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2ce7387-1f69-4770-8939-902c30a71e61-manager-config\") pod \"lws-controller-manager-8456f466bb-25cmx\" (UID: \"a2ce7387-1f69-4770-8939-902c30a71e61\") " pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:33.207508 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.207370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2ce7387-1f69-4770-8939-902c30a71e61-cert\") pod \"lws-controller-manager-8456f466bb-25cmx\" (UID: \"a2ce7387-1f69-4770-8939-902c30a71e61\") " pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:33.207508 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.207426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2ce7387-1f69-4770-8939-902c30a71e61-metrics-cert\") pod \"lws-controller-manager-8456f466bb-25cmx\" (UID: \"a2ce7387-1f69-4770-8939-902c30a71e61\") " pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:33.207962 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.207932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2ce7387-1f69-4770-8939-902c30a71e61-manager-config\") pod \"lws-controller-manager-8456f466bb-25cmx\" (UID: \"a2ce7387-1f69-4770-8939-902c30a71e61\") " pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:33.209775 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.209753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2ce7387-1f69-4770-8939-902c30a71e61-metrics-cert\") pod \"lws-controller-manager-8456f466bb-25cmx\" (UID: \"a2ce7387-1f69-4770-8939-902c30a71e61\") " pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:33.209930 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.209909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2ce7387-1f69-4770-8939-902c30a71e61-cert\") pod \"lws-controller-manager-8456f466bb-25cmx\" (UID: \"a2ce7387-1f69-4770-8939-902c30a71e61\") " pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:33.215676 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.215649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72jfd\" (UniqueName: \"kubernetes.io/projected/a2ce7387-1f69-4770-8939-902c30a71e61-kube-api-access-72jfd\") pod \"lws-controller-manager-8456f466bb-25cmx\" (UID: \"a2ce7387-1f69-4770-8939-902c30a71e61\") " pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:33.291964 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.291900    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:33.419508 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.419471    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"]
Jul 24 18:45:33.423213 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:45:33.423181    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ce7387_1f69_4770_8939_902c30a71e61.slice/crio-6c0402451db21a6be4fe7d660947e86354919b9c8fe1560d5b2c5e6781212d38 WatchSource:0}: Error finding container 6c0402451db21a6be4fe7d660947e86354919b9c8fe1560d5b2c5e6781212d38: Status 404 returned error can't find the container with id 6c0402451db21a6be4fe7d660947e86354919b9c8fe1560d5b2c5e6781212d38
Jul 24 18:45:33.535670 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:33.535639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx" event={"ID":"a2ce7387-1f69-4770-8939-902c30a71e61","Type":"ContainerStarted","Data":"6c0402451db21a6be4fe7d660947e86354919b9c8fe1560d5b2c5e6781212d38"}
Jul 24 18:45:36.550227 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:36.550194    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx" event={"ID":"a2ce7387-1f69-4770-8939-902c30a71e61","Type":"ContainerStarted","Data":"ce7958e1e30d90d11dc016fd9bb54c928934e93649dd23a4088c72eea0429f33"}
Jul 24 18:45:36.550623 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:36.550327    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:36.583359 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:36.583318    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx" podStartSLOduration=2.465186982 podStartE2EDuration="4.583304261s" totalImagesPullingTime="2.118117279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:45:36.581493886 +0000 UTC m=+335.957153182" watchObservedRunningTime="2026-07-24 18:45:36.583304261 +0000 UTC m=+335.958963557"
Jul 24 18:45:46.066603 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.066569    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq"]
Jul 24 18:45:46.070014 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.069990    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq"
Jul 24 18:45:46.072658 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.072633    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 24 18:45:46.072764 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.072676    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-tgqd5\""
Jul 24 18:45:46.072993 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.072974    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 24 18:45:46.073109 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.073014    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 18:45:46.073288 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.073269    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 18:45:46.087647 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.087623    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq"]
Jul 24 18:45:46.218438 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.218374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7j6c\" (UniqueName: \"kubernetes.io/projected/8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd-kube-api-access-z7j6c\") pod \"kube-auth-proxy-bbbd7cd9b-8wkrq\" (UID: \"8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd\") " pod="openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq"
Jul 24 18:45:46.218596 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.218451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd-tls-certs\") pod \"kube-auth-proxy-bbbd7cd9b-8wkrq\" (UID: \"8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd\") " pod="openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq"
Jul 24 18:45:46.218596 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.218553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd-tmp\") pod \"kube-auth-proxy-bbbd7cd9b-8wkrq\" (UID: \"8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd\") " pod="openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq"
Jul 24 18:45:46.319893 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.319815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd-tls-certs\") pod \"kube-auth-proxy-bbbd7cd9b-8wkrq\" (UID: \"8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd\") " pod="openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq"
Jul 24 18:45:46.320019 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.319912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd-tmp\") pod \"kube-auth-proxy-bbbd7cd9b-8wkrq\" (UID: \"8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd\") " pod="openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq"
Jul 24 18:45:46.320019 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.319969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7j6c\" (UniqueName: \"kubernetes.io/projected/8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd-kube-api-access-z7j6c\") pod \"kube-auth-proxy-bbbd7cd9b-8wkrq\" (UID: \"8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd\") " pod="openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq"
Jul 24 18:45:46.322017 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.321994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd-tmp\") pod \"kube-auth-proxy-bbbd7cd9b-8wkrq\" (UID: \"8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd\") " pod="openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq"
Jul 24 18:45:46.322297 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.322279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd-tls-certs\") pod \"kube-auth-proxy-bbbd7cd9b-8wkrq\" (UID: \"8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd\") " pod="openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq"
Jul 24 18:45:46.330423 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.330401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7j6c\" (UniqueName: \"kubernetes.io/projected/8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd-kube-api-access-z7j6c\") pod \"kube-auth-proxy-bbbd7cd9b-8wkrq\" (UID: \"8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd\") " pod="openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq"
Jul 24 18:45:46.379762 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.379743    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq"
Jul 24 18:45:46.516758 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.516731    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq"]
Jul 24 18:45:46.519493 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:45:46.519462    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1bca43_c0dc_4c6c_88d7_a945c7c95bfd.slice/crio-220c22249881b974f1825957efbb8d2db0571aed171d9954f832fbfb84e294eb WatchSource:0}: Error finding container 220c22249881b974f1825957efbb8d2db0571aed171d9954f832fbfb84e294eb: Status 404 returned error can't find the container with id 220c22249881b974f1825957efbb8d2db0571aed171d9954f832fbfb84e294eb
Jul 24 18:45:46.597755 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:46.597672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq" event={"ID":"8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd","Type":"ContainerStarted","Data":"220c22249881b974f1825957efbb8d2db0571aed171d9954f832fbfb84e294eb"}
Jul 24 18:45:47.556721 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:47.556693    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8456f466bb-25cmx"
Jul 24 18:45:49.613997 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:49.613960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq" event={"ID":"8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd","Type":"ContainerStarted","Data":"952ca2278ae7c14c481b4549ce85d7623e06c8e68e76e7d992e6eee6dc197c48"}
Jul 24 18:45:49.638518 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:45:49.638473    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-bbbd7cd9b-8wkrq" podStartSLOduration=0.954894155 podStartE2EDuration="3.638457714s" totalImagesPullingTime="2.683563559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:45:49.634355379 +0000 UTC m=+349.010014674" watchObservedRunningTime="2026-07-24 18:45:49.638457714 +0000 UTC m=+349.014117009"
Jul 24 18:47:29.023061 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.022985    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"]
Jul 24 18:47:29.026256 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.026237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"
Jul 24 18:47:29.029950 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.029920    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 24 18:47:29.030123 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.030106    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 18:47:29.030476 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.030455    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 18:47:29.030557 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.030489    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 24 18:47:29.030602 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.030497    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-49g7j\""
Jul 24 18:47:29.042415 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.042370    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"]
Jul 24 18:47:29.119285 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.119260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1cd9e89c-1fc9-487c-a346-9179f1e74d13-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-x5x4z\" (UID: \"1cd9e89c-1fc9-487c-a346-9179f1e74d13\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"
Jul 24 18:47:29.119421 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.119297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cd9e89c-1fc9-487c-a346-9179f1e74d13-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-x5x4z\" (UID: \"1cd9e89c-1fc9-487c-a346-9179f1e74d13\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"
Jul 24 18:47:29.119421 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.119399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr8kn\" (UniqueName: \"kubernetes.io/projected/1cd9e89c-1fc9-487c-a346-9179f1e74d13-kube-api-access-gr8kn\") pod \"kuadrant-console-plugin-66b5774985-x5x4z\" (UID: \"1cd9e89c-1fc9-487c-a346-9179f1e74d13\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"
Jul 24 18:47:29.219736 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.219713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr8kn\" (UniqueName: \"kubernetes.io/projected/1cd9e89c-1fc9-487c-a346-9179f1e74d13-kube-api-access-gr8kn\") pod \"kuadrant-console-plugin-66b5774985-x5x4z\" (UID: \"1cd9e89c-1fc9-487c-a346-9179f1e74d13\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"
Jul 24 18:47:29.219869 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.219754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1cd9e89c-1fc9-487c-a346-9179f1e74d13-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-x5x4z\" (UID: \"1cd9e89c-1fc9-487c-a346-9179f1e74d13\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"
Jul 24 18:47:29.219869 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.219786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cd9e89c-1fc9-487c-a346-9179f1e74d13-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-x5x4z\" (UID: \"1cd9e89c-1fc9-487c-a346-9179f1e74d13\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"
Jul 24 18:47:29.219947 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:47:29.219872    2580 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 24 18:47:29.219977 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:47:29.219951    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd9e89c-1fc9-487c-a346-9179f1e74d13-plugin-serving-cert podName:1cd9e89c-1fc9-487c-a346-9179f1e74d13 nodeName:}" failed. No retries permitted until 2026-07-24 18:47:29.719930871 +0000 UTC m=+449.095590164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/1cd9e89c-1fc9-487c-a346-9179f1e74d13-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-x5x4z" (UID: "1cd9e89c-1fc9-487c-a346-9179f1e74d13") : secret "plugin-serving-cert" not found
Jul 24 18:47:29.220400 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.220368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1cd9e89c-1fc9-487c-a346-9179f1e74d13-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-x5x4z\" (UID: \"1cd9e89c-1fc9-487c-a346-9179f1e74d13\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"
Jul 24 18:47:29.228467 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.228444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr8kn\" (UniqueName: \"kubernetes.io/projected/1cd9e89c-1fc9-487c-a346-9179f1e74d13-kube-api-access-gr8kn\") pod \"kuadrant-console-plugin-66b5774985-x5x4z\" (UID: \"1cd9e89c-1fc9-487c-a346-9179f1e74d13\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"
Jul 24 18:47:29.724192 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.724154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cd9e89c-1fc9-487c-a346-9179f1e74d13-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-x5x4z\" (UID: \"1cd9e89c-1fc9-487c-a346-9179f1e74d13\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"
Jul 24 18:47:29.726622 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.726590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cd9e89c-1fc9-487c-a346-9179f1e74d13-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-x5x4z\" (UID: \"1cd9e89c-1fc9-487c-a346-9179f1e74d13\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"
Jul 24 18:47:29.935524 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:29.935490    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"
Jul 24 18:47:30.067369 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:30.067339    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z"]
Jul 24 18:47:30.069481 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:47:30.069454    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cd9e89c_1fc9_487c_a346_9179f1e74d13.slice/crio-6b33db4ff7bd44bc5303885930dbaf2d1d71e7c040877ab24c9ea63d110fd283 WatchSource:0}: Error finding container 6b33db4ff7bd44bc5303885930dbaf2d1d71e7c040877ab24c9ea63d110fd283: Status 404 returned error can't find the container with id 6b33db4ff7bd44bc5303885930dbaf2d1d71e7c040877ab24c9ea63d110fd283
Jul 24 18:47:31.051167 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:31.049714    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z" event={"ID":"1cd9e89c-1fc9-487c-a346-9179f1e74d13","Type":"ContainerStarted","Data":"6b33db4ff7bd44bc5303885930dbaf2d1d71e7c040877ab24c9ea63d110fd283"}
Jul 24 18:47:38.338655 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:38.338618    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"]
Jul 24 18:47:38.345175 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:38.345151    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:38.348229 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:38.348209    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wm5k8\""
Jul 24 18:47:38.371230 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:38.371207    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"]
Jul 24 18:47:38.401486 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:38.401453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j5nr\" (UniqueName: \"kubernetes.io/projected/711bc2d6-b99c-4e8b-a6e0-292082b11e40-kube-api-access-6j5nr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dqd5v\" (UID: \"711bc2d6-b99c-4e8b-a6e0-292082b11e40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:38.401626 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:38.401502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/711bc2d6-b99c-4e8b-a6e0-292082b11e40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dqd5v\" (UID: \"711bc2d6-b99c-4e8b-a6e0-292082b11e40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:38.502571 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:38.502532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j5nr\" (UniqueName: \"kubernetes.io/projected/711bc2d6-b99c-4e8b-a6e0-292082b11e40-kube-api-access-6j5nr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dqd5v\" (UID: \"711bc2d6-b99c-4e8b-a6e0-292082b11e40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:38.502743 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:38.502579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/711bc2d6-b99c-4e8b-a6e0-292082b11e40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dqd5v\" (UID: \"711bc2d6-b99c-4e8b-a6e0-292082b11e40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:38.502986 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:38.502963    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/711bc2d6-b99c-4e8b-a6e0-292082b11e40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dqd5v\" (UID: \"711bc2d6-b99c-4e8b-a6e0-292082b11e40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:38.512648 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:38.512614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j5nr\" (UniqueName: \"kubernetes.io/projected/711bc2d6-b99c-4e8b-a6e0-292082b11e40-kube-api-access-6j5nr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dqd5v\" (UID: \"711bc2d6-b99c-4e8b-a6e0-292082b11e40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:38.657918 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:38.657830    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:39.067605 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:39.067546    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"]
Jul 24 18:47:39.076479 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:39.076441    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"]
Jul 24 18:47:39.090606 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:39.090573    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"]
Jul 24 18:47:39.097054 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:39.097032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"
Jul 24 18:47:39.109134 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:39.109107    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"]
Jul 24 18:47:39.207899 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:39.207867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b06a7be-fc5f-4362-8f78-16bbcb635ea9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-z5fwr\" (UID: \"1b06a7be-fc5f-4362-8f78-16bbcb635ea9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"
Jul 24 18:47:39.208087 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:39.207940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l9zr\" (UniqueName: \"kubernetes.io/projected/1b06a7be-fc5f-4362-8f78-16bbcb635ea9-kube-api-access-7l9zr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-z5fwr\" (UID: \"1b06a7be-fc5f-4362-8f78-16bbcb635ea9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"
Jul 24 18:47:39.308930 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:39.308893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l9zr\" (UniqueName: \"kubernetes.io/projected/1b06a7be-fc5f-4362-8f78-16bbcb635ea9-kube-api-access-7l9zr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-z5fwr\" (UID: \"1b06a7be-fc5f-4362-8f78-16bbcb635ea9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"
Jul 24 18:47:39.309103 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:39.308956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b06a7be-fc5f-4362-8f78-16bbcb635ea9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-z5fwr\" (UID: \"1b06a7be-fc5f-4362-8f78-16bbcb635ea9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"
Jul 24 18:47:39.309355 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:39.309324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b06a7be-fc5f-4362-8f78-16bbcb635ea9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-z5fwr\" (UID: \"1b06a7be-fc5f-4362-8f78-16bbcb635ea9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"
Jul 24 18:47:39.320342 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:39.320287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l9zr\" (UniqueName: \"kubernetes.io/projected/1b06a7be-fc5f-4362-8f78-16bbcb635ea9-kube-api-access-7l9zr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-z5fwr\" (UID: \"1b06a7be-fc5f-4362-8f78-16bbcb635ea9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"
Jul 24 18:47:39.413221 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:39.413185    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"
Jul 24 18:47:52.285237 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:52.285176    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"]
Jul 24 18:47:52.461393 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:47:52.461307    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b06a7be_fc5f_4362_8f78_16bbcb635ea9.slice/crio-97179598bf57b1bba47eed7a45c15fe56cc076a477ebbea4fe803536b46f0b19 WatchSource:0}: Error finding container 97179598bf57b1bba47eed7a45c15fe56cc076a477ebbea4fe803536b46f0b19: Status 404 returned error can't find the container with id 97179598bf57b1bba47eed7a45c15fe56cc076a477ebbea4fe803536b46f0b19
Jul 24 18:47:53.173765 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:53.173727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z" event={"ID":"1cd9e89c-1fc9-487c-a346-9179f1e74d13","Type":"ContainerStarted","Data":"348747a59b150ca17a6f75003fbbe8ca884ed641acd539543b31023060905086"}
Jul 24 18:47:53.174923 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:53.174899    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr" event={"ID":"1b06a7be-fc5f-4362-8f78-16bbcb635ea9","Type":"ContainerStarted","Data":"97179598bf57b1bba47eed7a45c15fe56cc076a477ebbea4fe803536b46f0b19"}
Jul 24 18:47:53.195288 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:53.195247    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x5x4z" podStartSLOduration=1.753766113 podStartE2EDuration="24.195232897s" totalImagesPullingTime="22.441466784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:47:53.194502983 +0000 UTC m=+472.570162281" watchObservedRunningTime="2026-07-24 18:47:53.195232897 +0000 UTC m=+472.570892193"
Jul 24 18:47:54.690751 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:47:54.690697    2580 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-dqd5v_kuadrant-system_711bc2d6-b99c-4e8b-a6e0-292082b11e40_0(23d66eca9bddca01fddd01ca94b8b86e306ad035dd67be6b060dcfe3c3ceaef5): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-dqd5v to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Jul 24 18:47:54.691113 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:47:54.690784    2580 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-dqd5v_kuadrant-system_711bc2d6-b99c-4e8b-a6e0-292082b11e40_0(23d66eca9bddca01fddd01ca94b8b86e306ad035dd67be6b060dcfe3c3ceaef5): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-dqd5v to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:55.185181 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:55.184844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:55.186925 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:47:55.186840    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-dqd5v\" is forbidden: User \"system:node:ip-10-0-128-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-111.ec2.internal' and this object" podUID="711bc2d6-b99c-4e8b-a6e0-292082b11e40" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:55.222514 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:55.222487    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:55.224252 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:47:55.224227    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-dqd5v\" is forbidden: User \"system:node:ip-10-0-128-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-111.ec2.internal' and this object" podUID="711bc2d6-b99c-4e8b-a6e0-292082b11e40" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:55.359876 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:55.359845    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j5nr\" (UniqueName: \"kubernetes.io/projected/711bc2d6-b99c-4e8b-a6e0-292082b11e40-kube-api-access-6j5nr\") pod \"711bc2d6-b99c-4e8b-a6e0-292082b11e40\" (UID: \"711bc2d6-b99c-4e8b-a6e0-292082b11e40\") "
Jul 24 18:47:55.360034 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:55.359889    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/711bc2d6-b99c-4e8b-a6e0-292082b11e40-extensions-socket-volume\") pod \"711bc2d6-b99c-4e8b-a6e0-292082b11e40\" (UID: \"711bc2d6-b99c-4e8b-a6e0-292082b11e40\") "
Jul 24 18:47:55.360207 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:55.360178    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711bc2d6-b99c-4e8b-a6e0-292082b11e40-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "711bc2d6-b99c-4e8b-a6e0-292082b11e40" (UID: "711bc2d6-b99c-4e8b-a6e0-292082b11e40"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:47:55.362166 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:55.362141    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711bc2d6-b99c-4e8b-a6e0-292082b11e40-kube-api-access-6j5nr" (OuterVolumeSpecName: "kube-api-access-6j5nr") pod "711bc2d6-b99c-4e8b-a6e0-292082b11e40" (UID: "711bc2d6-b99c-4e8b-a6e0-292082b11e40"). InnerVolumeSpecName "kube-api-access-6j5nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:47:55.461412 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:55.461321    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j5nr\" (UniqueName: \"kubernetes.io/projected/711bc2d6-b99c-4e8b-a6e0-292082b11e40-kube-api-access-6j5nr\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:47:55.461412 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:55.461353    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/711bc2d6-b99c-4e8b-a6e0-292082b11e40-extensions-socket-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:47:56.189485 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:56.189454    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:56.191741 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:47:56.191704    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-dqd5v\" is forbidden: User \"system:node:ip-10-0-128-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-111.ec2.internal' and this object" podUID="711bc2d6-b99c-4e8b-a6e0-292082b11e40" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:56.197507 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:47:56.197479    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-dqd5v\" is forbidden: User \"system:node:ip-10-0-128-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-111.ec2.internal' and this object" podUID="711bc2d6-b99c-4e8b-a6e0-292082b11e40" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dqd5v"
Jul 24 18:47:57.196985 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:57.196954    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="711bc2d6-b99c-4e8b-a6e0-292082b11e40" path="/var/lib/kubelet/pods/711bc2d6-b99c-4e8b-a6e0-292082b11e40/volumes"
Jul 24 18:47:59.205250 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:59.205216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr" event={"ID":"1b06a7be-fc5f-4362-8f78-16bbcb635ea9","Type":"ContainerStarted","Data":"d72bf7c4c8ccf1fc1381778f40dcfc1bccb16c17cb44660edff9516f1b38261e"}
Jul 24 18:47:59.205711 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:59.205346    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"
Jul 24 18:47:59.224827 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:47:59.224785    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr" podStartSLOduration=14.188309102 podStartE2EDuration="20.224771863s" totalImagesPullingTime="6.036462761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:47:59.224503577 +0000 UTC m=+478.600162883" watchObservedRunningTime="2026-07-24 18:47:59.224771863 +0000 UTC m=+478.600431160"
Jul 24 18:48:10.212896 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:10.212860    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"
Jul 24 18:48:22.676529 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:22.676496    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"]
Jul 24 18:48:22.677033 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:22.676756    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr" podUID="1b06a7be-fc5f-4362-8f78-16bbcb635ea9" containerName="manager" containerID="cri-o://d72bf7c4c8ccf1fc1381778f40dcfc1bccb16c17cb44660edff9516f1b38261e" gracePeriod=10
Jul 24 18:48:22.946697 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:22.946641    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"
Jul 24 18:48:23.082681 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.082647    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b06a7be-fc5f-4362-8f78-16bbcb635ea9-extensions-socket-volume\") pod \"1b06a7be-fc5f-4362-8f78-16bbcb635ea9\" (UID: \"1b06a7be-fc5f-4362-8f78-16bbcb635ea9\") "
Jul 24 18:48:23.082864 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.082693    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l9zr\" (UniqueName: \"kubernetes.io/projected/1b06a7be-fc5f-4362-8f78-16bbcb635ea9-kube-api-access-7l9zr\") pod \"1b06a7be-fc5f-4362-8f78-16bbcb635ea9\" (UID: \"1b06a7be-fc5f-4362-8f78-16bbcb635ea9\") "
Jul 24 18:48:23.082999 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.082975    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b06a7be-fc5f-4362-8f78-16bbcb635ea9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1b06a7be-fc5f-4362-8f78-16bbcb635ea9" (UID: "1b06a7be-fc5f-4362-8f78-16bbcb635ea9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:48:23.084774 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.084750    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b06a7be-fc5f-4362-8f78-16bbcb635ea9-kube-api-access-7l9zr" (OuterVolumeSpecName: "kube-api-access-7l9zr") pod "1b06a7be-fc5f-4362-8f78-16bbcb635ea9" (UID: "1b06a7be-fc5f-4362-8f78-16bbcb635ea9"). InnerVolumeSpecName "kube-api-access-7l9zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:48:23.184190 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.184164    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b06a7be-fc5f-4362-8f78-16bbcb635ea9-extensions-socket-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:48:23.184190 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.184187    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l9zr\" (UniqueName: \"kubernetes.io/projected/1b06a7be-fc5f-4362-8f78-16bbcb635ea9-kube-api-access-7l9zr\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:48:23.310057 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.310028    2580 generic.go:356] "Generic (PLEG): container finished" podID="1b06a7be-fc5f-4362-8f78-16bbcb635ea9" containerID="d72bf7c4c8ccf1fc1381778f40dcfc1bccb16c17cb44660edff9516f1b38261e" exitCode=0
Jul 24 18:48:23.310210 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.310072    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"
Jul 24 18:48:23.310210 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.310078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr" event={"ID":"1b06a7be-fc5f-4362-8f78-16bbcb635ea9","Type":"ContainerDied","Data":"d72bf7c4c8ccf1fc1381778f40dcfc1bccb16c17cb44660edff9516f1b38261e"}
Jul 24 18:48:23.310210 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.310136    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr" event={"ID":"1b06a7be-fc5f-4362-8f78-16bbcb635ea9","Type":"ContainerDied","Data":"97179598bf57b1bba47eed7a45c15fe56cc076a477ebbea4fe803536b46f0b19"}
Jul 24 18:48:23.310210 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.310161    2580 scope.go:117] "RemoveContainer" containerID="d72bf7c4c8ccf1fc1381778f40dcfc1bccb16c17cb44660edff9516f1b38261e"
Jul 24 18:48:23.318951 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.318933    2580 scope.go:117] "RemoveContainer" containerID="d72bf7c4c8ccf1fc1381778f40dcfc1bccb16c17cb44660edff9516f1b38261e"
Jul 24 18:48:23.319219 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:48:23.319197    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72bf7c4c8ccf1fc1381778f40dcfc1bccb16c17cb44660edff9516f1b38261e\": container with ID starting with d72bf7c4c8ccf1fc1381778f40dcfc1bccb16c17cb44660edff9516f1b38261e not found: ID does not exist" containerID="d72bf7c4c8ccf1fc1381778f40dcfc1bccb16c17cb44660edff9516f1b38261e"
Jul 24 18:48:23.319265 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.319229    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72bf7c4c8ccf1fc1381778f40dcfc1bccb16c17cb44660edff9516f1b38261e"} err="failed to get container status \"d72bf7c4c8ccf1fc1381778f40dcfc1bccb16c17cb44660edff9516f1b38261e\": rpc error: code = NotFound desc = could not find container \"d72bf7c4c8ccf1fc1381778f40dcfc1bccb16c17cb44660edff9516f1b38261e\": container with ID starting with d72bf7c4c8ccf1fc1381778f40dcfc1bccb16c17cb44660edff9516f1b38261e not found: ID does not exist"
Jul 24 18:48:23.331647 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.331626    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"]
Jul 24 18:48:23.336116 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:23.336097    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-z5fwr"]
Jul 24 18:48:25.196497 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:25.196465    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b06a7be-fc5f-4362-8f78-16bbcb635ea9" path="/var/lib/kubelet/pods/1b06a7be-fc5f-4362-8f78-16bbcb635ea9/volumes"
Jul 24 18:48:29.020942 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.020911    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"]
Jul 24 18:48:29.021408 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.021224    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b06a7be-fc5f-4362-8f78-16bbcb635ea9" containerName="manager"
Jul 24 18:48:29.021408 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.021235    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b06a7be-fc5f-4362-8f78-16bbcb635ea9" containerName="manager"
Jul 24 18:48:29.021408 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.021287    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b06a7be-fc5f-4362-8f78-16bbcb635ea9" containerName="manager"
Jul 24 18:48:29.024416 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.024400    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"
Jul 24 18:48:29.026503 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.026481    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 18:48:29.033335 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.033314    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"]
Jul 24 18:48:29.119283 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.119253    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"]
Jul 24 18:48:29.131440 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.131411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbzz6\" (UniqueName: \"kubernetes.io/projected/4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87-kube-api-access-qbzz6\") pod \"limitador-limitador-64c94fd64f-x2h9r\" (UID: \"4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"
Jul 24 18:48:29.131582 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.131515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87-config-file\") pod \"limitador-limitador-64c94fd64f-x2h9r\" (UID: \"4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"
Jul 24 18:48:29.232893 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.232868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbzz6\" (UniqueName: \"kubernetes.io/projected/4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87-kube-api-access-qbzz6\") pod \"limitador-limitador-64c94fd64f-x2h9r\" (UID: \"4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"
Jul 24 18:48:29.233039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.232921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87-config-file\") pod \"limitador-limitador-64c94fd64f-x2h9r\" (UID: \"4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"
Jul 24 18:48:29.233484 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.233466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87-config-file\") pod \"limitador-limitador-64c94fd64f-x2h9r\" (UID: \"4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"
Jul 24 18:48:29.240904 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.240879    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbzz6\" (UniqueName: \"kubernetes.io/projected/4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87-kube-api-access-qbzz6\") pod \"limitador-limitador-64c94fd64f-x2h9r\" (UID: \"4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"
Jul 24 18:48:29.334124 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.334085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"
Jul 24 18:48:29.466342 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:29.466317    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"]
Jul 24 18:48:29.469036 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:48:29.469001    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6fb94f_acc9_45b1_bfe6_bd0c6210ad87.slice/crio-c31b504e7d9d6f234010013c159fa12b098b85965799314a2e4521bae2b89d9b WatchSource:0}: Error finding container c31b504e7d9d6f234010013c159fa12b098b85965799314a2e4521bae2b89d9b: Status 404 returned error can't find the container with id c31b504e7d9d6f234010013c159fa12b098b85965799314a2e4521bae2b89d9b
Jul 24 18:48:30.340968 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:30.340936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r" event={"ID":"4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87","Type":"ContainerStarted","Data":"c31b504e7d9d6f234010013c159fa12b098b85965799314a2e4521bae2b89d9b"}
Jul 24 18:48:33.358028 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:33.357990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r" event={"ID":"4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87","Type":"ContainerStarted","Data":"f3e11f2677f9c2d91cce17aef3c094c24ecf8c70e13d7abf1c3eab6fe88e6a14"}
Jul 24 18:48:33.358419 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:33.358116    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"
Jul 24 18:48:33.375069 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:33.375023    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r" podStartSLOduration=1.170046828 podStartE2EDuration="4.375008704s" totalImagesPullingTime="3.204961876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:48:33.373016928 +0000 UTC m=+512.748676333" watchObservedRunningTime="2026-07-24 18:48:33.375008704 +0000 UTC m=+512.750668000"
Jul 24 18:48:44.126008 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:44.125970    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"]
Jul 24 18:48:44.126548 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:44.126165    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r" podUID="4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87" containerName="limitador" containerID="cri-o://f3e11f2677f9c2d91cce17aef3c094c24ecf8c70e13d7abf1c3eab6fe88e6a14" gracePeriod=30
Jul 24 18:48:44.126943 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:44.126843    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"
Jul 24 18:48:45.059290 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.059264    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"
Jul 24 18:48:45.166423 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.166371    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87-config-file\") pod \"4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87\" (UID: \"4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87\") "
Jul 24 18:48:45.166764 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.166460    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbzz6\" (UniqueName: \"kubernetes.io/projected/4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87-kube-api-access-qbzz6\") pod \"4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87\" (UID: \"4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87\") "
Jul 24 18:48:45.166764 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.166712    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87-config-file" (OuterVolumeSpecName: "config-file") pod "4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87" (UID: "4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:48:45.168420 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.168398    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87-kube-api-access-qbzz6" (OuterVolumeSpecName: "kube-api-access-qbzz6") pod "4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87" (UID: "4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87"). InnerVolumeSpecName "kube-api-access-qbzz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:48:45.267332 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.267301    2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87-config-file\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:48:45.267332 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.267328    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbzz6\" (UniqueName: \"kubernetes.io/projected/4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87-kube-api-access-qbzz6\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:48:45.407523 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.407490    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87" containerID="f3e11f2677f9c2d91cce17aef3c094c24ecf8c70e13d7abf1c3eab6fe88e6a14" exitCode=0
Jul 24 18:48:45.407644 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.407538    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"
Jul 24 18:48:45.407644 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.407580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r" event={"ID":"4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87","Type":"ContainerDied","Data":"f3e11f2677f9c2d91cce17aef3c094c24ecf8c70e13d7abf1c3eab6fe88e6a14"}
Jul 24 18:48:45.407644 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.407634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2h9r" event={"ID":"4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87","Type":"ContainerDied","Data":"c31b504e7d9d6f234010013c159fa12b098b85965799314a2e4521bae2b89d9b"}
Jul 24 18:48:45.407747 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.407656    2580 scope.go:117] "RemoveContainer" containerID="f3e11f2677f9c2d91cce17aef3c094c24ecf8c70e13d7abf1c3eab6fe88e6a14"
Jul 24 18:48:45.415566 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.415549    2580 scope.go:117] "RemoveContainer" containerID="f3e11f2677f9c2d91cce17aef3c094c24ecf8c70e13d7abf1c3eab6fe88e6a14"
Jul 24 18:48:45.415824 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:48:45.415804    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e11f2677f9c2d91cce17aef3c094c24ecf8c70e13d7abf1c3eab6fe88e6a14\": container with ID starting with f3e11f2677f9c2d91cce17aef3c094c24ecf8c70e13d7abf1c3eab6fe88e6a14 not found: ID does not exist" containerID="f3e11f2677f9c2d91cce17aef3c094c24ecf8c70e13d7abf1c3eab6fe88e6a14"
Jul 24 18:48:45.415881 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.415832    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e11f2677f9c2d91cce17aef3c094c24ecf8c70e13d7abf1c3eab6fe88e6a14"} err="failed to get container status \"f3e11f2677f9c2d91cce17aef3c094c24ecf8c70e13d7abf1c3eab6fe88e6a14\": rpc error: code = NotFound desc = could not find container \"f3e11f2677f9c2d91cce17aef3c094c24ecf8c70e13d7abf1c3eab6fe88e6a14\": container with ID starting with f3e11f2677f9c2d91cce17aef3c094c24ecf8c70e13d7abf1c3eab6fe88e6a14 not found: ID does not exist"
Jul 24 18:48:45.426848 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.426818    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"]
Jul 24 18:48:45.430950 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:45.430929    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x2h9r"]
Jul 24 18:48:47.196736 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:47.196703    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87" path="/var/lib/kubelet/pods/4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87/volumes"
Jul 24 18:48:50.521718 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.521687    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-w24m2"]
Jul 24 18:48:50.522135 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.522003    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87" containerName="limitador"
Jul 24 18:48:50.522135 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.522014    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87" containerName="limitador"
Jul 24 18:48:50.522135 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.522075    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a6fb94f-acc9-45b1-bfe6-bd0c6210ad87" containerName="limitador"
Jul 24 18:48:50.526329 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.526309    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-w24m2"
Jul 24 18:48:50.528792 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.528770    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:48:50.529127 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.529110    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:48:50.529232 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.529169    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 24 18:48:50.529335 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.529229    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dxgnb\""
Jul 24 18:48:50.540524 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.540503    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-w24m2"]
Jul 24 18:48:50.707462 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.707432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5-data\") pod \"postgres-69fb4594cf-w24m2\" (UID: \"d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-w24m2"
Jul 24 18:48:50.707607 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.707479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8pvm\" (UniqueName: \"kubernetes.io/projected/d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5-kube-api-access-f8pvm\") pod \"postgres-69fb4594cf-w24m2\" (UID: \"d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-w24m2"
Jul 24 18:48:50.808270 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.808186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8pvm\" (UniqueName: \"kubernetes.io/projected/d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5-kube-api-access-f8pvm\") pod \"postgres-69fb4594cf-w24m2\" (UID: \"d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-w24m2"
Jul 24 18:48:50.808446 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.808286    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5-data\") pod \"postgres-69fb4594cf-w24m2\" (UID: \"d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-w24m2"
Jul 24 18:48:50.808683 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.808661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5-data\") pod \"postgres-69fb4594cf-w24m2\" (UID: \"d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-w24m2"
Jul 24 18:48:50.825582 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.825553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8pvm\" (UniqueName: \"kubernetes.io/projected/d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5-kube-api-access-f8pvm\") pod \"postgres-69fb4594cf-w24m2\" (UID: \"d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-w24m2"
Jul 24 18:48:50.836388 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.836356    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-w24m2"
Jul 24 18:48:50.969287 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:50.969254    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-w24m2"]
Jul 24 18:48:50.973088 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:48:50.973057    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79dca2c_fa23_441a_8a18_f5b6fc0ab9f5.slice/crio-32257b6d18e483cc4308c32f52167f73120b98e94cf3a0a44766a16e6f680d60 WatchSource:0}: Error finding container 32257b6d18e483cc4308c32f52167f73120b98e94cf3a0a44766a16e6f680d60: Status 404 returned error can't find the container with id 32257b6d18e483cc4308c32f52167f73120b98e94cf3a0a44766a16e6f680d60
Jul 24 18:48:51.435973 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:51.435940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-w24m2" event={"ID":"d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5","Type":"ContainerStarted","Data":"32257b6d18e483cc4308c32f52167f73120b98e94cf3a0a44766a16e6f680d60"}
Jul 24 18:48:59.478863 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:59.478825    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-w24m2" event={"ID":"d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5","Type":"ContainerStarted","Data":"4197de00cdaa1b7da5e04429f19c99759b289377ff3b596a5433ee267fef8f33"}
Jul 24 18:48:59.479259 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:48:59.478903    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-w24m2"
Jul 24 18:49:05.511400 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:05.511353    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-w24m2"
Jul 24 18:49:05.528447 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:05.528407    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-w24m2" podStartSLOduration=7.76188745 podStartE2EDuration="15.528391489s" totalImagesPullingTime="7.766504039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:48:59.523821797 +0000 UTC m=+538.899481093" watchObservedRunningTime="2026-07-24 18:49:05.528391489 +0000 UTC m=+544.904050776"
Jul 24 18:49:12.599884 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.599853    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cf5c69646-sbk7n"]
Jul 24 18:49:12.606942 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.606921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cf5c69646-sbk7n"
Jul 24 18:49:12.608899 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.608875    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 18:49:12.609019 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.608901    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-22s4l\""
Jul 24 18:49:12.609019 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.608921    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 18:49:12.613037 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.613013    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cf5c69646-sbk7n"]
Jul 24 18:49:12.686932 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.686900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcsrx\" (UniqueName: \"kubernetes.io/projected/6467879a-509a-43d6-9923-88cec0586b65-kube-api-access-wcsrx\") pod \"maas-controller-6cf5c69646-sbk7n\" (UID: \"6467879a-509a-43d6-9923-88cec0586b65\") " pod="opendatahub/maas-controller-6cf5c69646-sbk7n"
Jul 24 18:49:12.687086 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.687018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6467879a-509a-43d6-9923-88cec0586b65-webhook-cert\") pod \"maas-controller-6cf5c69646-sbk7n\" (UID: \"6467879a-509a-43d6-9923-88cec0586b65\") " pod="opendatahub/maas-controller-6cf5c69646-sbk7n"
Jul 24 18:49:12.759476 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.759437    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64bbdb967c-7wrzl"]
Jul 24 18:49:12.762793 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.762774    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64bbdb967c-7wrzl"
Jul 24 18:49:12.773268 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.773242    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64bbdb967c-7wrzl"]
Jul 24 18:49:12.787507 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.787487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcsrx\" (UniqueName: \"kubernetes.io/projected/6467879a-509a-43d6-9923-88cec0586b65-kube-api-access-wcsrx\") pod \"maas-controller-6cf5c69646-sbk7n\" (UID: \"6467879a-509a-43d6-9923-88cec0586b65\") " pod="opendatahub/maas-controller-6cf5c69646-sbk7n"
Jul 24 18:49:12.787640 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.787524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d38948cf-819c-4b54-bd31-b94a4bf2c032-webhook-cert\") pod \"maas-controller-64bbdb967c-7wrzl\" (UID: \"d38948cf-819c-4b54-bd31-b94a4bf2c032\") " pod="opendatahub/maas-controller-64bbdb967c-7wrzl"
Jul 24 18:49:12.787640 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.787554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5db5\" (UniqueName: \"kubernetes.io/projected/d38948cf-819c-4b54-bd31-b94a4bf2c032-kube-api-access-b5db5\") pod \"maas-controller-64bbdb967c-7wrzl\" (UID: \"d38948cf-819c-4b54-bd31-b94a4bf2c032\") " pod="opendatahub/maas-controller-64bbdb967c-7wrzl"
Jul 24 18:49:12.787640 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.787592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6467879a-509a-43d6-9923-88cec0586b65-webhook-cert\") pod \"maas-controller-6cf5c69646-sbk7n\" (UID: \"6467879a-509a-43d6-9923-88cec0586b65\") " pod="opendatahub/maas-controller-6cf5c69646-sbk7n"
Jul 24 18:49:12.787795 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:49:12.787719    2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 24 18:49:12.787855 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:49:12.787802    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6467879a-509a-43d6-9923-88cec0586b65-webhook-cert podName:6467879a-509a-43d6-9923-88cec0586b65 nodeName:}" failed. No retries permitted until 2026-07-24 18:49:13.287773693 +0000 UTC m=+552.663432970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/6467879a-509a-43d6-9923-88cec0586b65-webhook-cert") pod "maas-controller-6cf5c69646-sbk7n" (UID: "6467879a-509a-43d6-9923-88cec0586b65") : secret "maas-controller-webhook-cert" not found
Jul 24 18:49:12.795373 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.795351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcsrx\" (UniqueName: \"kubernetes.io/projected/6467879a-509a-43d6-9923-88cec0586b65-kube-api-access-wcsrx\") pod \"maas-controller-6cf5c69646-sbk7n\" (UID: \"6467879a-509a-43d6-9923-88cec0586b65\") " pod="opendatahub/maas-controller-6cf5c69646-sbk7n"
Jul 24 18:49:12.888548 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.888488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5db5\" (UniqueName: \"kubernetes.io/projected/d38948cf-819c-4b54-bd31-b94a4bf2c032-kube-api-access-b5db5\") pod \"maas-controller-64bbdb967c-7wrzl\" (UID: \"d38948cf-819c-4b54-bd31-b94a4bf2c032\") " pod="opendatahub/maas-controller-64bbdb967c-7wrzl"
Jul 24 18:49:12.888651 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.888624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d38948cf-819c-4b54-bd31-b94a4bf2c032-webhook-cert\") pod \"maas-controller-64bbdb967c-7wrzl\" (UID: \"d38948cf-819c-4b54-bd31-b94a4bf2c032\") " pod="opendatahub/maas-controller-64bbdb967c-7wrzl"
Jul 24 18:49:12.888755 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:49:12.888740    2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 24 18:49:12.888819 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:49:12.888806    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38948cf-819c-4b54-bd31-b94a4bf2c032-webhook-cert podName:d38948cf-819c-4b54-bd31-b94a4bf2c032 nodeName:}" failed. No retries permitted until 2026-07-24 18:49:13.388781967 +0000 UTC m=+552.764441255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/d38948cf-819c-4b54-bd31-b94a4bf2c032-webhook-cert") pod "maas-controller-64bbdb967c-7wrzl" (UID: "d38948cf-819c-4b54-bd31-b94a4bf2c032") : secret "maas-controller-webhook-cert" not found
Jul 24 18:49:12.899674 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:12.899651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5db5\" (UniqueName: \"kubernetes.io/projected/d38948cf-819c-4b54-bd31-b94a4bf2c032-kube-api-access-b5db5\") pod \"maas-controller-64bbdb967c-7wrzl\" (UID: \"d38948cf-819c-4b54-bd31-b94a4bf2c032\") " pod="opendatahub/maas-controller-64bbdb967c-7wrzl"
Jul 24 18:49:13.292111 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:13.292025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6467879a-509a-43d6-9923-88cec0586b65-webhook-cert\") pod \"maas-controller-6cf5c69646-sbk7n\" (UID: \"6467879a-509a-43d6-9923-88cec0586b65\") " pod="opendatahub/maas-controller-6cf5c69646-sbk7n"
Jul 24 18:49:13.294505 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:13.294479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6467879a-509a-43d6-9923-88cec0586b65-webhook-cert\") pod \"maas-controller-6cf5c69646-sbk7n\" (UID: \"6467879a-509a-43d6-9923-88cec0586b65\") " pod="opendatahub/maas-controller-6cf5c69646-sbk7n"
Jul 24 18:49:13.393348 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:13.393317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d38948cf-819c-4b54-bd31-b94a4bf2c032-webhook-cert\") pod \"maas-controller-64bbdb967c-7wrzl\" (UID: \"d38948cf-819c-4b54-bd31-b94a4bf2c032\") " pod="opendatahub/maas-controller-64bbdb967c-7wrzl"
Jul 24 18:49:13.395516 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:13.395495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d38948cf-819c-4b54-bd31-b94a4bf2c032-webhook-cert\") pod \"maas-controller-64bbdb967c-7wrzl\" (UID: \"d38948cf-819c-4b54-bd31-b94a4bf2c032\") " pod="opendatahub/maas-controller-64bbdb967c-7wrzl"
Jul 24 18:49:13.518344 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:13.518318    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cf5c69646-sbk7n"
Jul 24 18:49:13.644986 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:13.644938    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cf5c69646-sbk7n"]
Jul 24 18:49:13.648109 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:49:13.648085    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6467879a_509a_43d6_9923_88cec0586b65.slice/crio-bd6238c8f824f16c5affb0b3473f8d0c41ce3ecf2f3a924b26e12d68d33401e2 WatchSource:0}: Error finding container bd6238c8f824f16c5affb0b3473f8d0c41ce3ecf2f3a924b26e12d68d33401e2: Status 404 returned error can't find the container with id bd6238c8f824f16c5affb0b3473f8d0c41ce3ecf2f3a924b26e12d68d33401e2
Jul 24 18:49:13.673646 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:13.673624    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64bbdb967c-7wrzl"
Jul 24 18:49:13.798821 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:13.798788    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64bbdb967c-7wrzl"]
Jul 24 18:49:13.802095 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:49:13.802071    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38948cf_819c_4b54_bd31_b94a4bf2c032.slice/crio-c4879b0e5a69a24d4c275686d2bccb37060497726f93112671b52ef1e081744e WatchSource:0}: Error finding container c4879b0e5a69a24d4c275686d2bccb37060497726f93112671b52ef1e081744e: Status 404 returned error can't find the container with id c4879b0e5a69a24d4c275686d2bccb37060497726f93112671b52ef1e081744e
Jul 24 18:49:14.556884 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:14.556843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64bbdb967c-7wrzl" event={"ID":"d38948cf-819c-4b54-bd31-b94a4bf2c032","Type":"ContainerStarted","Data":"c4879b0e5a69a24d4c275686d2bccb37060497726f93112671b52ef1e081744e"}
Jul 24 18:49:14.558936 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:14.558906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cf5c69646-sbk7n" event={"ID":"6467879a-509a-43d6-9923-88cec0586b65","Type":"ContainerStarted","Data":"bd6238c8f824f16c5affb0b3473f8d0c41ce3ecf2f3a924b26e12d68d33401e2"}
Jul 24 18:49:16.573566 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:16.573527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64bbdb967c-7wrzl" event={"ID":"d38948cf-819c-4b54-bd31-b94a4bf2c032","Type":"ContainerStarted","Data":"08085a43a69ccf168bdbbb010ee0b37a27368b11c27da763d60607e2985d5299"}
Jul 24 18:49:16.574042 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:16.573643    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64bbdb967c-7wrzl"
Jul 24 18:49:16.575202 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:16.575178    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cf5c69646-sbk7n" event={"ID":"6467879a-509a-43d6-9923-88cec0586b65","Type":"ContainerStarted","Data":"16d1a01047dfe247bf9ec2b112ab4e6e7b622bf48a206d5f9760464e20d0a4a6"}
Jul 24 18:49:16.575311 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:16.575300    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cf5c69646-sbk7n"
Jul 24 18:49:16.611148 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:16.611084    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-64bbdb967c-7wrzl" podStartSLOduration=2.1699539469999998 podStartE2EDuration="4.611062961s" totalImagesPullingTime="2.441109014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:49:16.609890205 +0000 UTC m=+555.985549491" watchObservedRunningTime="2026-07-24 18:49:16.611062961 +0000 UTC m=+555.986722270"
Jul 24 18:49:16.657368 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:16.657302    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cf5c69646-sbk7n" podStartSLOduration=2.055120979 podStartE2EDuration="4.657281351s" totalImagesPullingTime="2.602160372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:49:16.655127077 +0000 UTC m=+556.030786375" watchObservedRunningTime="2026-07-24 18:49:16.657281351 +0000 UTC m=+556.032940648"
Jul 24 18:49:27.581393 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:27.581345    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cf5c69646-sbk7n"
Jul 24 18:49:27.581774 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:27.581441    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64bbdb967c-7wrzl"
Jul 24 18:49:27.640545 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:27.640508    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cf5c69646-sbk7n"]
Jul 24 18:49:27.640702 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:27.640668    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6cf5c69646-sbk7n" podUID="6467879a-509a-43d6-9923-88cec0586b65" containerName="manager" containerID="cri-o://16d1a01047dfe247bf9ec2b112ab4e6e7b622bf48a206d5f9760464e20d0a4a6" gracePeriod=30
Jul 24 18:49:27.878369 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:27.878345    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cf5c69646-sbk7n"
Jul 24 18:49:28.020223 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.020196    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcsrx\" (UniqueName: \"kubernetes.io/projected/6467879a-509a-43d6-9923-88cec0586b65-kube-api-access-wcsrx\") pod \"6467879a-509a-43d6-9923-88cec0586b65\" (UID: \"6467879a-509a-43d6-9923-88cec0586b65\") "
Jul 24 18:49:28.020400 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.020265    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6467879a-509a-43d6-9923-88cec0586b65-webhook-cert\") pod \"6467879a-509a-43d6-9923-88cec0586b65\" (UID: \"6467879a-509a-43d6-9923-88cec0586b65\") "
Jul 24 18:49:28.022333 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.022304    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6467879a-509a-43d6-9923-88cec0586b65-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6467879a-509a-43d6-9923-88cec0586b65" (UID: "6467879a-509a-43d6-9923-88cec0586b65"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:49:28.022457 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.022351    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6467879a-509a-43d6-9923-88cec0586b65-kube-api-access-wcsrx" (OuterVolumeSpecName: "kube-api-access-wcsrx") pod "6467879a-509a-43d6-9923-88cec0586b65" (UID: "6467879a-509a-43d6-9923-88cec0586b65"). InnerVolumeSpecName "kube-api-access-wcsrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:49:28.121501 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.121439    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcsrx\" (UniqueName: \"kubernetes.io/projected/6467879a-509a-43d6-9923-88cec0586b65-kube-api-access-wcsrx\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:49:28.121501 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.121463    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6467879a-509a-43d6-9923-88cec0586b65-webhook-cert\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:49:28.627667 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.627632    2580 generic.go:356] "Generic (PLEG): container finished" podID="6467879a-509a-43d6-9923-88cec0586b65" containerID="16d1a01047dfe247bf9ec2b112ab4e6e7b622bf48a206d5f9760464e20d0a4a6" exitCode=0
Jul 24 18:49:28.627667 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.627667    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cf5c69646-sbk7n"
Jul 24 18:49:28.628129 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.627699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cf5c69646-sbk7n" event={"ID":"6467879a-509a-43d6-9923-88cec0586b65","Type":"ContainerDied","Data":"16d1a01047dfe247bf9ec2b112ab4e6e7b622bf48a206d5f9760464e20d0a4a6"}
Jul 24 18:49:28.628129 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.627744    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cf5c69646-sbk7n" event={"ID":"6467879a-509a-43d6-9923-88cec0586b65","Type":"ContainerDied","Data":"bd6238c8f824f16c5affb0b3473f8d0c41ce3ecf2f3a924b26e12d68d33401e2"}
Jul 24 18:49:28.628129 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.627768    2580 scope.go:117] "RemoveContainer" containerID="16d1a01047dfe247bf9ec2b112ab4e6e7b622bf48a206d5f9760464e20d0a4a6"
Jul 24 18:49:28.638224 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.638201    2580 scope.go:117] "RemoveContainer" containerID="16d1a01047dfe247bf9ec2b112ab4e6e7b622bf48a206d5f9760464e20d0a4a6"
Jul 24 18:49:28.638520 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:49:28.638502    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d1a01047dfe247bf9ec2b112ab4e6e7b622bf48a206d5f9760464e20d0a4a6\": container with ID starting with 16d1a01047dfe247bf9ec2b112ab4e6e7b622bf48a206d5f9760464e20d0a4a6 not found: ID does not exist" containerID="16d1a01047dfe247bf9ec2b112ab4e6e7b622bf48a206d5f9760464e20d0a4a6"
Jul 24 18:49:28.638597 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.638531    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d1a01047dfe247bf9ec2b112ab4e6e7b622bf48a206d5f9760464e20d0a4a6"} err="failed to get container status \"16d1a01047dfe247bf9ec2b112ab4e6e7b622bf48a206d5f9760464e20d0a4a6\": rpc error: code = NotFound desc = could not find container \"16d1a01047dfe247bf9ec2b112ab4e6e7b622bf48a206d5f9760464e20d0a4a6\": container with ID starting with 16d1a01047dfe247bf9ec2b112ab4e6e7b622bf48a206d5f9760464e20d0a4a6 not found: ID does not exist"
Jul 24 18:49:28.652217 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.651913    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cf5c69646-sbk7n"]
Jul 24 18:49:28.657138 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:28.657111    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cf5c69646-sbk7n"]
Jul 24 18:49:29.196806 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:29.196772    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6467879a-509a-43d6-9923-88cec0586b65" path="/var/lib/kubelet/pods/6467879a-509a-43d6-9923-88cec0586b65/volumes"
Jul 24 18:49:47.001185 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.001146    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"]
Jul 24 18:49:47.001715 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.001550    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6467879a-509a-43d6-9923-88cec0586b65" containerName="manager"
Jul 24 18:49:47.001715 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.001566    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6467879a-509a-43d6-9923-88cec0586b65" containerName="manager"
Jul 24 18:49:47.001715 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.001637    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6467879a-509a-43d6-9923-88cec0586b65" containerName="manager"
Jul 24 18:49:47.029223 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.029200    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"]
Jul 24 18:49:47.029357 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.029288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:49:47.031681 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.031653    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z489t\""
Jul 24 18:49:47.032154 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.032132    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 24 18:49:47.178909 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.178873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ebc295-677f-4143-9882-a569761a46cd-maas-api-tls\") pod \"maas-api-5bbf9886d8-kn7md\" (UID: \"07ebc295-677f-4143-9882-a569761a46cd\") " pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:49:47.179066 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.178986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbssr\" (UniqueName: \"kubernetes.io/projected/07ebc295-677f-4143-9882-a569761a46cd-kube-api-access-gbssr\") pod \"maas-api-5bbf9886d8-kn7md\" (UID: \"07ebc295-677f-4143-9882-a569761a46cd\") " pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:49:47.179066 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.179042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ebc295-677f-4143-9882-a569761a46cd-openshift-service-ca\") pod \"maas-api-5bbf9886d8-kn7md\" (UID: \"07ebc295-677f-4143-9882-a569761a46cd\") " pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:49:47.279738 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.279670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbssr\" (UniqueName: \"kubernetes.io/projected/07ebc295-677f-4143-9882-a569761a46cd-kube-api-access-gbssr\") pod \"maas-api-5bbf9886d8-kn7md\" (UID: \"07ebc295-677f-4143-9882-a569761a46cd\") " pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:49:47.279738 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.279730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ebc295-677f-4143-9882-a569761a46cd-openshift-service-ca\") pod \"maas-api-5bbf9886d8-kn7md\" (UID: \"07ebc295-677f-4143-9882-a569761a46cd\") " pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:49:47.279941 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.279757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ebc295-677f-4143-9882-a569761a46cd-maas-api-tls\") pod \"maas-api-5bbf9886d8-kn7md\" (UID: \"07ebc295-677f-4143-9882-a569761a46cd\") " pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:49:47.279941 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:49:47.279872    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 24 18:49:47.280052 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:49:47.279950    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07ebc295-677f-4143-9882-a569761a46cd-maas-api-tls podName:07ebc295-677f-4143-9882-a569761a46cd nodeName:}" failed. No retries permitted until 2026-07-24 18:49:47.779928251 +0000 UTC m=+587.155587531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07ebc295-677f-4143-9882-a569761a46cd-maas-api-tls") pod "maas-api-5bbf9886d8-kn7md" (UID: "07ebc295-677f-4143-9882-a569761a46cd") : secret "maas-api-serving-cert" not found
Jul 24 18:49:47.280542 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.280521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ebc295-677f-4143-9882-a569761a46cd-openshift-service-ca\") pod \"maas-api-5bbf9886d8-kn7md\" (UID: \"07ebc295-677f-4143-9882-a569761a46cd\") " pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:49:47.292530 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.292510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbssr\" (UniqueName: \"kubernetes.io/projected/07ebc295-677f-4143-9882-a569761a46cd-kube-api-access-gbssr\") pod \"maas-api-5bbf9886d8-kn7md\" (UID: \"07ebc295-677f-4143-9882-a569761a46cd\") " pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:49:47.567071 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.567034    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq"]
Jul 24 18:49:47.570608 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.570587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq"
Jul 24 18:49:47.575244 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.575227    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 24 18:49:47.575469 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.575448    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5l9ww\""
Jul 24 18:49:47.582004 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.581983    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq"]
Jul 24 18:49:47.592892 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.592866    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-pz5wb"]
Jul 24 18:49:47.596208 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.596174    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-pz5wb"
Jul 24 18:49:47.608557 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.608534    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-pz5wb"]
Jul 24 18:49:47.683512 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.683482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f271893-54fc-4c88-b0dd-6b5affc95805-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-w44xq\" (UID: \"6f271893-54fc-4c88-b0dd-6b5affc95805\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq"
Jul 24 18:49:47.683695 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.683520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7x8d\" (UniqueName: \"kubernetes.io/projected/6f271893-54fc-4c88-b0dd-6b5affc95805-kube-api-access-x7x8d\") pod \"payload-pre-processing-687dbb6cf6-w44xq\" (UID: \"6f271893-54fc-4c88-b0dd-6b5affc95805\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq"
Jul 24 18:49:47.683695 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.683537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02328292-4ed8-4caf-a880-594f6920e321-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-pz5wb\" (UID: \"02328292-4ed8-4caf-a880-594f6920e321\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pz5wb"
Jul 24 18:49:47.683695 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.683663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq22f\" (UniqueName: \"kubernetes.io/projected/02328292-4ed8-4caf-a880-594f6920e321-kube-api-access-sq22f\") pod \"payload-processing-5cc6cd444b-pz5wb\" (UID: \"02328292-4ed8-4caf-a880-594f6920e321\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pz5wb"
Jul 24 18:49:47.784732 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.784698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f271893-54fc-4c88-b0dd-6b5affc95805-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-w44xq\" (UID: \"6f271893-54fc-4c88-b0dd-6b5affc95805\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq"
Jul 24 18:49:47.784732 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.784736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7x8d\" (UniqueName: \"kubernetes.io/projected/6f271893-54fc-4c88-b0dd-6b5affc95805-kube-api-access-x7x8d\") pod \"payload-pre-processing-687dbb6cf6-w44xq\" (UID: \"6f271893-54fc-4c88-b0dd-6b5affc95805\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq"
Jul 24 18:49:47.784966 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.784753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02328292-4ed8-4caf-a880-594f6920e321-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-pz5wb\" (UID: \"02328292-4ed8-4caf-a880-594f6920e321\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pz5wb"
Jul 24 18:49:47.784966 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.784775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ebc295-677f-4143-9882-a569761a46cd-maas-api-tls\") pod \"maas-api-5bbf9886d8-kn7md\" (UID: \"07ebc295-677f-4143-9882-a569761a46cd\") " pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:49:47.784966 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.784804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq22f\" (UniqueName: \"kubernetes.io/projected/02328292-4ed8-4caf-a880-594f6920e321-kube-api-access-sq22f\") pod \"payload-processing-5cc6cd444b-pz5wb\" (UID: \"02328292-4ed8-4caf-a880-594f6920e321\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pz5wb"
Jul 24 18:49:47.784966 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:49:47.784910    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 24 18:49:47.785175 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:49:47.784981    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07ebc295-677f-4143-9882-a569761a46cd-maas-api-tls podName:07ebc295-677f-4143-9882-a569761a46cd nodeName:}" failed. No retries permitted until 2026-07-24 18:49:48.784962356 +0000 UTC m=+588.160621634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07ebc295-677f-4143-9882-a569761a46cd-maas-api-tls") pod "maas-api-5bbf9886d8-kn7md" (UID: "07ebc295-677f-4143-9882-a569761a46cd") : secret "maas-api-serving-cert" not found
Jul 24 18:49:47.785529 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.785506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02328292-4ed8-4caf-a880-594f6920e321-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-pz5wb\" (UID: \"02328292-4ed8-4caf-a880-594f6920e321\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pz5wb"
Jul 24 18:49:47.785621 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.785546    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f271893-54fc-4c88-b0dd-6b5affc95805-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-w44xq\" (UID: \"6f271893-54fc-4c88-b0dd-6b5affc95805\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq"
Jul 24 18:49:47.792649 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.792623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7x8d\" (UniqueName: \"kubernetes.io/projected/6f271893-54fc-4c88-b0dd-6b5affc95805-kube-api-access-x7x8d\") pod \"payload-pre-processing-687dbb6cf6-w44xq\" (UID: \"6f271893-54fc-4c88-b0dd-6b5affc95805\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq"
Jul 24 18:49:47.793097 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.793076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq22f\" (UniqueName: \"kubernetes.io/projected/02328292-4ed8-4caf-a880-594f6920e321-kube-api-access-sq22f\") pod \"payload-processing-5cc6cd444b-pz5wb\" (UID: \"02328292-4ed8-4caf-a880-594f6920e321\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pz5wb"
Jul 24 18:49:47.881743 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.881678    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq"
Jul 24 18:49:47.906178 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:47.906149    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-pz5wb"
Jul 24 18:49:48.023668 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:48.023638    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq"]
Jul 24 18:49:48.027443 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:49:48.027412    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f271893_54fc_4c88_b0dd_6b5affc95805.slice/crio-0278c5a842daf8999b2a0ada38d2bbdb600e8983b0083f33baeda98a93a9b093 WatchSource:0}: Error finding container 0278c5a842daf8999b2a0ada38d2bbdb600e8983b0083f33baeda98a93a9b093: Status 404 returned error can't find the container with id 0278c5a842daf8999b2a0ada38d2bbdb600e8983b0083f33baeda98a93a9b093
Jul 24 18:49:48.057406 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:48.057312    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-pz5wb"]
Jul 24 18:49:48.059535 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:49:48.059513    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02328292_4ed8_4caf_a880_594f6920e321.slice/crio-51d29e556fdae0f5a31f0273daafa212e54404f71fcb796b10f2aa9ad53e6ec2 WatchSource:0}: Error finding container 51d29e556fdae0f5a31f0273daafa212e54404f71fcb796b10f2aa9ad53e6ec2: Status 404 returned error can't find the container with id 51d29e556fdae0f5a31f0273daafa212e54404f71fcb796b10f2aa9ad53e6ec2
Jul 24 18:49:48.720245 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:48.720210    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-pz5wb" event={"ID":"02328292-4ed8-4caf-a880-594f6920e321","Type":"ContainerStarted","Data":"51d29e556fdae0f5a31f0273daafa212e54404f71fcb796b10f2aa9ad53e6ec2"}
Jul 24 18:49:48.722325 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:48.722274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq" event={"ID":"6f271893-54fc-4c88-b0dd-6b5affc95805","Type":"ContainerStarted","Data":"0278c5a842daf8999b2a0ada38d2bbdb600e8983b0083f33baeda98a93a9b093"}
Jul 24 18:49:48.793698 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:48.793664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ebc295-677f-4143-9882-a569761a46cd-maas-api-tls\") pod \"maas-api-5bbf9886d8-kn7md\" (UID: \"07ebc295-677f-4143-9882-a569761a46cd\") " pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:49:48.799000 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:48.798948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07ebc295-677f-4143-9882-a569761a46cd-maas-api-tls\") pod \"maas-api-5bbf9886d8-kn7md\" (UID: \"07ebc295-677f-4143-9882-a569761a46cd\") " pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:49:48.839137 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:48.838844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:49:49.051715 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:49.051139    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"]
Jul 24 18:49:49.055364 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:49:49.055323    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ebc295_677f_4143_9882_a569761a46cd.slice/crio-46111597a319c21f3f1dd25bf4e6e7d18c83990a233a8e3ec2afe33c15fdd95f WatchSource:0}: Error finding container 46111597a319c21f3f1dd25bf4e6e7d18c83990a233a8e3ec2afe33c15fdd95f: Status 404 returned error can't find the container with id 46111597a319c21f3f1dd25bf4e6e7d18c83990a233a8e3ec2afe33c15fdd95f
Jul 24 18:49:49.729692 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:49.729657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md" event={"ID":"07ebc295-677f-4143-9882-a569761a46cd","Type":"ContainerStarted","Data":"46111597a319c21f3f1dd25bf4e6e7d18c83990a233a8e3ec2afe33c15fdd95f"}
Jul 24 18:49:50.738256 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:50.738151    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-pz5wb" event={"ID":"02328292-4ed8-4caf-a880-594f6920e321","Type":"ContainerStarted","Data":"764b79022de6a0d3df002d86e9ca720b0d1f4f864c8146817531cbdfbf9ec476"}
Jul 24 18:49:50.738256 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:50.738246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-pz5wb"
Jul 24 18:49:50.740492 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:50.740448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq" event={"ID":"6f271893-54fc-4c88-b0dd-6b5affc95805","Type":"ContainerStarted","Data":"a1179752fed8e2fd061d209b0b1149f8a16c61695c7a6e76ce99a00221891a28"}
Jul 24 18:49:50.740642 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:50.740624    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq"
Jul 24 18:49:50.756048 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:50.755991    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-pz5wb" podStartSLOduration=1.422767951 podStartE2EDuration="3.755971958s" totalImagesPullingTime="2.333204007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:49:50.753896751 +0000 UTC m=+590.129556048" watchObservedRunningTime="2026-07-24 18:49:50.755971958 +0000 UTC m=+590.131631238"
Jul 24 18:49:50.772699 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:50.772655    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq" podStartSLOduration=1.401971079 podStartE2EDuration="3.772643098s" totalImagesPullingTime="2.370672019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:49:50.771040917 +0000 UTC m=+590.146700214" watchObservedRunningTime="2026-07-24 18:49:50.772643098 +0000 UTC m=+590.148302397"
Jul 24 18:49:51.747030 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:51.746995    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md" event={"ID":"07ebc295-677f-4143-9882-a569761a46cd","Type":"ContainerStarted","Data":"b34c022e71155df4df6bf082b84f7e945e8836e8329aca44379c418d31aa68a7"}
Jul 24 18:49:51.747473 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:51.747098    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:49:51.764450 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:51.764408    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md" podStartSLOduration=3.29456295 podStartE2EDuration="5.764395791s" totalImagesPullingTime="2.469832841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:49:51.763318292 +0000 UTC m=+591.138977598" watchObservedRunningTime="2026-07-24 18:49:51.764395791 +0000 UTC m=+591.140055079"
Jul 24 18:49:57.757356 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:49:57.757322    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bbf9886d8-kn7md"
Jul 24 18:50:01.069830 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:01.069802    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 18:50:01.085632 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:01.085605    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 18:50:01.749955 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:01.749926    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-pz5wb"
Jul 24 18:50:01.750124 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:01.750103    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w44xq"
Jul 24 18:50:14.441653 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.441622    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"]
Jul 24 18:50:14.446181 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.446161    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.448580 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.448560    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 18:50:14.448664 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.448561    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 18:50:14.449119 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.449101    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pq4pw\""
Jul 24 18:50:14.449197 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.449146    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 24 18:50:14.456349 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.456331    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"]
Jul 24 18:50:14.504691 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.504664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a572efa1-e59f-4f4f-8448-3517c4bf9e05-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.504806 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.504699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a572efa1-e59f-4f4f-8448-3517c4bf9e05-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.504806 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.504731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a572efa1-e59f-4f4f-8448-3517c4bf9e05-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.504806 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.504752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a572efa1-e59f-4f4f-8448-3517c4bf9e05-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.504806 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.504774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhmtp\" (UniqueName: \"kubernetes.io/projected/a572efa1-e59f-4f4f-8448-3517c4bf9e05-kube-api-access-fhmtp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.504806 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.504803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a572efa1-e59f-4f4f-8448-3517c4bf9e05-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.605089 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.605062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a572efa1-e59f-4f4f-8448-3517c4bf9e05-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.605204 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.605095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a572efa1-e59f-4f4f-8448-3517c4bf9e05-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.605204 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.605122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a572efa1-e59f-4f4f-8448-3517c4bf9e05-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.605204 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.605143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a572efa1-e59f-4f4f-8448-3517c4bf9e05-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.605204 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.605164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhmtp\" (UniqueName: \"kubernetes.io/projected/a572efa1-e59f-4f4f-8448-3517c4bf9e05-kube-api-access-fhmtp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.605204 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.605192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a572efa1-e59f-4f4f-8448-3517c4bf9e05-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.605546 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.605517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a572efa1-e59f-4f4f-8448-3517c4bf9e05-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.605729 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.605568    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a572efa1-e59f-4f4f-8448-3517c4bf9e05-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.605729 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.605648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a572efa1-e59f-4f4f-8448-3517c4bf9e05-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.607457 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.607435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a572efa1-e59f-4f4f-8448-3517c4bf9e05-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.607710 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.607695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a572efa1-e59f-4f4f-8448-3517c4bf9e05-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.612497 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.612473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhmtp\" (UniqueName: \"kubernetes.io/projected/a572efa1-e59f-4f4f-8448-3517c4bf9e05-kube-api-access-fhmtp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf\" (UID: \"a572efa1-e59f-4f4f-8448-3517c4bf9e05\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.757200 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.757124    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:14.891011 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:14.890986    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"]
Jul 24 18:50:14.893271 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:50:14.893235    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda572efa1_e59f_4f4f_8448_3517c4bf9e05.slice/crio-158a091d2452d1a8d8273cf5c5d093e8c2ef702404cc3190992d84611516384e WatchSource:0}: Error finding container 158a091d2452d1a8d8273cf5c5d093e8c2ef702404cc3190992d84611516384e: Status 404 returned error can't find the container with id 158a091d2452d1a8d8273cf5c5d093e8c2ef702404cc3190992d84611516384e
Jul 24 18:50:15.864373 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:15.864336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf" event={"ID":"a572efa1-e59f-4f4f-8448-3517c4bf9e05","Type":"ContainerStarted","Data":"158a091d2452d1a8d8273cf5c5d093e8c2ef702404cc3190992d84611516384e"}
Jul 24 18:50:16.871069 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:16.871031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf" event={"ID":"a572efa1-e59f-4f4f-8448-3517c4bf9e05","Type":"ContainerStarted","Data":"288bb885051137157976bd54c1730f92491d3b00de5353eefe7399c96fef6bd0"}
Jul 24 18:50:16.888699 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:16.888647    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf" podStartSLOduration=1.737610273 podStartE2EDuration="2.888631763s" totalImagesPullingTime="1.15102149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:50:16.887163325 +0000 UTC m=+616.262822621" watchObservedRunningTime="2026-07-24 18:50:16.888631763 +0000 UTC m=+616.264291059"
Jul 24 18:50:24.758123 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:24.758045    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:24.758123 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:24.758107    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:24.770652 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:24.770626    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:24.917877 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:24.917843    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf"
Jul 24 18:50:56.060237 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.060197    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"]
Jul 24 18:50:56.063838 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.063816    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.068471 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.068448    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 24 18:50:56.084351 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.084315    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"]
Jul 24 18:50:56.149098 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.149067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a177f9-63e2-402c-93e4-374946bd82a1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.149233 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.149108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a177f9-63e2-402c-93e4-374946bd82a1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.149233 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.149130    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scnlk\" (UniqueName: \"kubernetes.io/projected/d1a177f9-63e2-402c-93e4-374946bd82a1-kube-api-access-scnlk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.149307 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.149246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a177f9-63e2-402c-93e4-374946bd82a1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.149307 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.149282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a177f9-63e2-402c-93e4-374946bd82a1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.149374 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.149332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a177f9-63e2-402c-93e4-374946bd82a1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.249964 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.249936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a177f9-63e2-402c-93e4-374946bd82a1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.250117 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.249977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a177f9-63e2-402c-93e4-374946bd82a1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.250117 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.250001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a177f9-63e2-402c-93e4-374946bd82a1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.250214 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.250126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scnlk\" (UniqueName: \"kubernetes.io/projected/d1a177f9-63e2-402c-93e4-374946bd82a1-kube-api-access-scnlk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.250274 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.250261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a177f9-63e2-402c-93e4-374946bd82a1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.250332 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.250298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a177f9-63e2-402c-93e4-374946bd82a1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.250561 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.250424    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a177f9-63e2-402c-93e4-374946bd82a1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.250561 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.250509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a177f9-63e2-402c-93e4-374946bd82a1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.250738 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.250704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a177f9-63e2-402c-93e4-374946bd82a1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.252129 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.252106    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a177f9-63e2-402c-93e4-374946bd82a1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.252774 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.252752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a177f9-63e2-402c-93e4-374946bd82a1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.261435 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.261414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scnlk\" (UniqueName: \"kubernetes.io/projected/d1a177f9-63e2-402c-93e4-374946bd82a1-kube-api-access-scnlk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sfl6n\" (UID: \"d1a177f9-63e2-402c-93e4-374946bd82a1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.382886 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.382849    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:50:56.514206 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.514180    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"]
Jul 24 18:50:56.516269 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:50:56.516246    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a177f9_63e2_402c_93e4_374946bd82a1.slice/crio-ba793e03e4f5389901b5eddfef12af416eda3b174387a6608d47ca445304e2f3 WatchSource:0}: Error finding container ba793e03e4f5389901b5eddfef12af416eda3b174387a6608d47ca445304e2f3: Status 404 returned error can't find the container with id ba793e03e4f5389901b5eddfef12af416eda3b174387a6608d47ca445304e2f3
Jul 24 18:50:56.517541 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:56.517524    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:50:57.052242 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:57.052153    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n" event={"ID":"d1a177f9-63e2-402c-93e4-374946bd82a1","Type":"ContainerStarted","Data":"ffc3381457d658f4890a553b1894af24bf49a691b6fcf1f090297f8e4ae9a368"}
Jul 24 18:50:57.052242 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:57.052221    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n" event={"ID":"d1a177f9-63e2-402c-93e4-374946bd82a1","Type":"ContainerStarted","Data":"ba793e03e4f5389901b5eddfef12af416eda3b174387a6608d47ca445304e2f3"}
Jul 24 18:50:57.076740 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:50:57.076698    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n" podStartSLOduration=0.881261566 podStartE2EDuration="1.076683851s" totalImagesPullingTime="195.422285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:50:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:50:57.076061459 +0000 UTC m=+656.451720755" watchObservedRunningTime="2026-07-24 18:50:57.076683851 +0000 UTC m=+656.452343147"
Jul 24 18:51:06.383733 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:51:06.383697    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:51:06.384167 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:51:06.383766    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:51:06.396160 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:51:06.396136    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:51:07.111211 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:51:07.111178    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sfl6n"
Jul 24 18:53:29.841177 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:29.841097    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64bbdb967c-7wrzl"]
Jul 24 18:53:29.841789 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:29.841307    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-64bbdb967c-7wrzl" podUID="d38948cf-819c-4b54-bd31-b94a4bf2c032" containerName="manager" containerID="cri-o://08085a43a69ccf168bdbbb010ee0b37a27368b11c27da763d60607e2985d5299" gracePeriod=30
Jul 24 18:53:30.087305 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.087275    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64bbdb967c-7wrzl"
Jul 24 18:53:30.217308 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.217224    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d38948cf-819c-4b54-bd31-b94a4bf2c032-webhook-cert\") pod \"d38948cf-819c-4b54-bd31-b94a4bf2c032\" (UID: \"d38948cf-819c-4b54-bd31-b94a4bf2c032\") "
Jul 24 18:53:30.217488 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.217321    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5db5\" (UniqueName: \"kubernetes.io/projected/d38948cf-819c-4b54-bd31-b94a4bf2c032-kube-api-access-b5db5\") pod \"d38948cf-819c-4b54-bd31-b94a4bf2c032\" (UID: \"d38948cf-819c-4b54-bd31-b94a4bf2c032\") "
Jul 24 18:53:30.219356 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.219322    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d38948cf-819c-4b54-bd31-b94a4bf2c032-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d38948cf-819c-4b54-bd31-b94a4bf2c032" (UID: "d38948cf-819c-4b54-bd31-b94a4bf2c032"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:53:30.219480 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.219351    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38948cf-819c-4b54-bd31-b94a4bf2c032-kube-api-access-b5db5" (OuterVolumeSpecName: "kube-api-access-b5db5") pod "d38948cf-819c-4b54-bd31-b94a4bf2c032" (UID: "d38948cf-819c-4b54-bd31-b94a4bf2c032"). InnerVolumeSpecName "kube-api-access-b5db5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:53:30.318749 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.318718    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5db5\" (UniqueName: \"kubernetes.io/projected/d38948cf-819c-4b54-bd31-b94a4bf2c032-kube-api-access-b5db5\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:53:30.318749 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.318746    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d38948cf-819c-4b54-bd31-b94a4bf2c032-webhook-cert\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:53:30.761680 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.761645    2580 generic.go:356] "Generic (PLEG): container finished" podID="d38948cf-819c-4b54-bd31-b94a4bf2c032" containerID="08085a43a69ccf168bdbbb010ee0b37a27368b11c27da763d60607e2985d5299" exitCode=0
Jul 24 18:53:30.761875 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.761693    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64bbdb967c-7wrzl"
Jul 24 18:53:30.761875 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.761730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64bbdb967c-7wrzl" event={"ID":"d38948cf-819c-4b54-bd31-b94a4bf2c032","Type":"ContainerDied","Data":"08085a43a69ccf168bdbbb010ee0b37a27368b11c27da763d60607e2985d5299"}
Jul 24 18:53:30.761875 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.761783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64bbdb967c-7wrzl" event={"ID":"d38948cf-819c-4b54-bd31-b94a4bf2c032","Type":"ContainerDied","Data":"c4879b0e5a69a24d4c275686d2bccb37060497726f93112671b52ef1e081744e"}
Jul 24 18:53:30.761875 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.761806    2580 scope.go:117] "RemoveContainer" containerID="08085a43a69ccf168bdbbb010ee0b37a27368b11c27da763d60607e2985d5299"
Jul 24 18:53:30.771057 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.771042    2580 scope.go:117] "RemoveContainer" containerID="08085a43a69ccf168bdbbb010ee0b37a27368b11c27da763d60607e2985d5299"
Jul 24 18:53:30.771333 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:53:30.771314    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08085a43a69ccf168bdbbb010ee0b37a27368b11c27da763d60607e2985d5299\": container with ID starting with 08085a43a69ccf168bdbbb010ee0b37a27368b11c27da763d60607e2985d5299 not found: ID does not exist" containerID="08085a43a69ccf168bdbbb010ee0b37a27368b11c27da763d60607e2985d5299"
Jul 24 18:53:30.771402 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.771341    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08085a43a69ccf168bdbbb010ee0b37a27368b11c27da763d60607e2985d5299"} err="failed to get container status \"08085a43a69ccf168bdbbb010ee0b37a27368b11c27da763d60607e2985d5299\": rpc error: code = NotFound desc = could not find container \"08085a43a69ccf168bdbbb010ee0b37a27368b11c27da763d60607e2985d5299\": container with ID starting with 08085a43a69ccf168bdbbb010ee0b37a27368b11c27da763d60607e2985d5299 not found: ID does not exist"
Jul 24 18:53:30.781633 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.781610    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64bbdb967c-7wrzl"]
Jul 24 18:53:30.786016 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:30.785996    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64bbdb967c-7wrzl"]
Jul 24 18:53:31.196403 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:53:31.196359    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d38948cf-819c-4b54-bd31-b94a4bf2c032" path="/var/lib/kubelet/pods/d38948cf-819c-4b54-bd31-b94a4bf2c032/volumes"
Jul 24 18:54:31.578634 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.578605    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64bbdb967c-8hmdb"]
Jul 24 18:54:31.579052 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.578974    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d38948cf-819c-4b54-bd31-b94a4bf2c032" containerName="manager"
Jul 24 18:54:31.579052 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.578985    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d38948cf-819c-4b54-bd31-b94a4bf2c032" containerName="manager"
Jul 24 18:54:31.579052 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.579051    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d38948cf-819c-4b54-bd31-b94a4bf2c032" containerName="manager"
Jul 24 18:54:31.582046 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.582025    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64bbdb967c-8hmdb"
Jul 24 18:54:31.584451 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.584434    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 18:54:31.584548 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.584433    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-22s4l\""
Jul 24 18:54:31.584548 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.584440    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 18:54:31.592173 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.592149    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64bbdb967c-8hmdb"]
Jul 24 18:54:31.714063 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.714029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a05ebe94-1eb7-4e49-b16e-d53df93b2666-webhook-cert\") pod \"maas-controller-64bbdb967c-8hmdb\" (UID: \"a05ebe94-1eb7-4e49-b16e-d53df93b2666\") " pod="opendatahub/maas-controller-64bbdb967c-8hmdb"
Jul 24 18:54:31.714213 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.714092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts422\" (UniqueName: \"kubernetes.io/projected/a05ebe94-1eb7-4e49-b16e-d53df93b2666-kube-api-access-ts422\") pod \"maas-controller-64bbdb967c-8hmdb\" (UID: \"a05ebe94-1eb7-4e49-b16e-d53df93b2666\") " pod="opendatahub/maas-controller-64bbdb967c-8hmdb"
Jul 24 18:54:31.814688 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.814655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a05ebe94-1eb7-4e49-b16e-d53df93b2666-webhook-cert\") pod \"maas-controller-64bbdb967c-8hmdb\" (UID: \"a05ebe94-1eb7-4e49-b16e-d53df93b2666\") " pod="opendatahub/maas-controller-64bbdb967c-8hmdb"
Jul 24 18:54:31.814862 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.814711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts422\" (UniqueName: \"kubernetes.io/projected/a05ebe94-1eb7-4e49-b16e-d53df93b2666-kube-api-access-ts422\") pod \"maas-controller-64bbdb967c-8hmdb\" (UID: \"a05ebe94-1eb7-4e49-b16e-d53df93b2666\") " pod="opendatahub/maas-controller-64bbdb967c-8hmdb"
Jul 24 18:54:31.817032 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.817004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a05ebe94-1eb7-4e49-b16e-d53df93b2666-webhook-cert\") pod \"maas-controller-64bbdb967c-8hmdb\" (UID: \"a05ebe94-1eb7-4e49-b16e-d53df93b2666\") " pod="opendatahub/maas-controller-64bbdb967c-8hmdb"
Jul 24 18:54:31.822356 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.822330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts422\" (UniqueName: \"kubernetes.io/projected/a05ebe94-1eb7-4e49-b16e-d53df93b2666-kube-api-access-ts422\") pod \"maas-controller-64bbdb967c-8hmdb\" (UID: \"a05ebe94-1eb7-4e49-b16e-d53df93b2666\") " pod="opendatahub/maas-controller-64bbdb967c-8hmdb"
Jul 24 18:54:31.893427 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:31.893339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64bbdb967c-8hmdb"
Jul 24 18:54:32.021227 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:32.021012    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64bbdb967c-8hmdb"]
Jul 24 18:54:32.023581 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:54:32.023556    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05ebe94_1eb7_4e49_b16e_d53df93b2666.slice/crio-3ffda049aca786c933336fed9038409ba1d8c454b528e7f44a24051f283de96f WatchSource:0}: Error finding container 3ffda049aca786c933336fed9038409ba1d8c454b528e7f44a24051f283de96f: Status 404 returned error can't find the container with id 3ffda049aca786c933336fed9038409ba1d8c454b528e7f44a24051f283de96f
Jul 24 18:54:32.033985 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:32.033951    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64bbdb967c-8hmdb" event={"ID":"a05ebe94-1eb7-4e49-b16e-d53df93b2666","Type":"ContainerStarted","Data":"3ffda049aca786c933336fed9038409ba1d8c454b528e7f44a24051f283de96f"}
Jul 24 18:54:33.040552 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:33.040511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64bbdb967c-8hmdb" event={"ID":"a05ebe94-1eb7-4e49-b16e-d53df93b2666","Type":"ContainerStarted","Data":"c794e983d933ce33d21ecaa84d5c64366f2d1e088406bbc0174fcb8d15d82f0b"}
Jul 24 18:54:33.040964 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:33.040652    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64bbdb967c-8hmdb"
Jul 24 18:54:33.057685 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:33.057637    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-64bbdb967c-8hmdb" podStartSLOduration=1.5953280859999999 podStartE2EDuration="2.057622231s" totalImagesPullingTime="462.294145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:54:33.055876593 +0000 UTC m=+872.431535888" watchObservedRunningTime="2026-07-24 18:54:33.057622231 +0000 UTC m=+872.433281521"
Jul 24 18:54:44.049943 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:44.049911    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64bbdb967c-8hmdb"
Jul 24 18:54:48.675628 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:48.675594    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64bbdb967c-8hmdb"]
Jul 24 18:54:48.676092 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:48.675814    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-64bbdb967c-8hmdb" podUID="a05ebe94-1eb7-4e49-b16e-d53df93b2666" containerName="manager" containerID="cri-o://c794e983d933ce33d21ecaa84d5c64366f2d1e088406bbc0174fcb8d15d82f0b" gracePeriod=30
Jul 24 18:54:48.916402 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:48.916361    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64bbdb967c-8hmdb"
Jul 24 18:54:49.055651 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.055579    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a05ebe94-1eb7-4e49-b16e-d53df93b2666-webhook-cert\") pod \"a05ebe94-1eb7-4e49-b16e-d53df93b2666\" (UID: \"a05ebe94-1eb7-4e49-b16e-d53df93b2666\") "
Jul 24 18:54:49.055778 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.055677    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts422\" (UniqueName: \"kubernetes.io/projected/a05ebe94-1eb7-4e49-b16e-d53df93b2666-kube-api-access-ts422\") pod \"a05ebe94-1eb7-4e49-b16e-d53df93b2666\" (UID: \"a05ebe94-1eb7-4e49-b16e-d53df93b2666\") "
Jul 24 18:54:49.057601 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.057572    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a05ebe94-1eb7-4e49-b16e-d53df93b2666-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a05ebe94-1eb7-4e49-b16e-d53df93b2666" (UID: "a05ebe94-1eb7-4e49-b16e-d53df93b2666"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:54:49.057712 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.057601    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05ebe94-1eb7-4e49-b16e-d53df93b2666-kube-api-access-ts422" (OuterVolumeSpecName: "kube-api-access-ts422") pod "a05ebe94-1eb7-4e49-b16e-d53df93b2666" (UID: "a05ebe94-1eb7-4e49-b16e-d53df93b2666"). InnerVolumeSpecName "kube-api-access-ts422". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:54:49.117444 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.117411    2580 generic.go:356] "Generic (PLEG): container finished" podID="a05ebe94-1eb7-4e49-b16e-d53df93b2666" containerID="c794e983d933ce33d21ecaa84d5c64366f2d1e088406bbc0174fcb8d15d82f0b" exitCode=0
Jul 24 18:54:49.117444 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.117445    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64bbdb967c-8hmdb" event={"ID":"a05ebe94-1eb7-4e49-b16e-d53df93b2666","Type":"ContainerDied","Data":"c794e983d933ce33d21ecaa84d5c64366f2d1e088406bbc0174fcb8d15d82f0b"}
Jul 24 18:54:49.117634 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.117465    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64bbdb967c-8hmdb"
Jul 24 18:54:49.117634 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.117506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64bbdb967c-8hmdb" event={"ID":"a05ebe94-1eb7-4e49-b16e-d53df93b2666","Type":"ContainerDied","Data":"3ffda049aca786c933336fed9038409ba1d8c454b528e7f44a24051f283de96f"}
Jul 24 18:54:49.117634 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.117528    2580 scope.go:117] "RemoveContainer" containerID="c794e983d933ce33d21ecaa84d5c64366f2d1e088406bbc0174fcb8d15d82f0b"
Jul 24 18:54:49.126624 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.126594    2580 scope.go:117] "RemoveContainer" containerID="c794e983d933ce33d21ecaa84d5c64366f2d1e088406bbc0174fcb8d15d82f0b"
Jul 24 18:54:49.126858 ip-10-0-128-111 kubenswrapper[2580]: E0724 18:54:49.126839    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c794e983d933ce33d21ecaa84d5c64366f2d1e088406bbc0174fcb8d15d82f0b\": container with ID starting with c794e983d933ce33d21ecaa84d5c64366f2d1e088406bbc0174fcb8d15d82f0b not found: ID does not exist" containerID="c794e983d933ce33d21ecaa84d5c64366f2d1e088406bbc0174fcb8d15d82f0b"
Jul 24 18:54:49.126919 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.126865    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c794e983d933ce33d21ecaa84d5c64366f2d1e088406bbc0174fcb8d15d82f0b"} err="failed to get container status \"c794e983d933ce33d21ecaa84d5c64366f2d1e088406bbc0174fcb8d15d82f0b\": rpc error: code = NotFound desc = could not find container \"c794e983d933ce33d21ecaa84d5c64366f2d1e088406bbc0174fcb8d15d82f0b\": container with ID starting with c794e983d933ce33d21ecaa84d5c64366f2d1e088406bbc0174fcb8d15d82f0b not found: ID does not exist"
Jul 24 18:54:49.138597 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.138570    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64bbdb967c-8hmdb"]
Jul 24 18:54:49.142106 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.142084    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64bbdb967c-8hmdb"]
Jul 24 18:54:49.157093 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.157070    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts422\" (UniqueName: \"kubernetes.io/projected/a05ebe94-1eb7-4e49-b16e-d53df93b2666-kube-api-access-ts422\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:54:49.157162 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.157093    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a05ebe94-1eb7-4e49-b16e-d53df93b2666-webhook-cert\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 18:54:49.196350 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.196327    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a05ebe94-1eb7-4e49-b16e-d53df93b2666" path="/var/lib/kubelet/pods/a05ebe94-1eb7-4e49-b16e-d53df93b2666/volumes"
Jul 24 18:54:49.905838 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.905807    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64bbdb967c-t228l"]
Jul 24 18:54:49.906210 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.906161    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a05ebe94-1eb7-4e49-b16e-d53df93b2666" containerName="manager"
Jul 24 18:54:49.906210 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.906172    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a05ebe94-1eb7-4e49-b16e-d53df93b2666" containerName="manager"
Jul 24 18:54:49.906295 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.906237    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a05ebe94-1eb7-4e49-b16e-d53df93b2666" containerName="manager"
Jul 24 18:54:49.910644 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.910625    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64bbdb967c-t228l"
Jul 24 18:54:49.913872 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.913849    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 18:54:49.914021 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.913849    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 18:54:49.914021 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.913897    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-22s4l\""
Jul 24 18:54:49.919765 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:49.919738    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64bbdb967c-t228l"]
Jul 24 18:54:50.065476 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:50.065451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6kds\" (UniqueName: \"kubernetes.io/projected/513c3d5f-ad2d-4852-ad67-bdbfc750fa42-kube-api-access-w6kds\") pod \"maas-controller-64bbdb967c-t228l\" (UID: \"513c3d5f-ad2d-4852-ad67-bdbfc750fa42\") " pod="opendatahub/maas-controller-64bbdb967c-t228l"
Jul 24 18:54:50.065618 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:50.065558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/513c3d5f-ad2d-4852-ad67-bdbfc750fa42-webhook-cert\") pod \"maas-controller-64bbdb967c-t228l\" (UID: \"513c3d5f-ad2d-4852-ad67-bdbfc750fa42\") " pod="opendatahub/maas-controller-64bbdb967c-t228l"
Jul 24 18:54:50.166638 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:50.166602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6kds\" (UniqueName: \"kubernetes.io/projected/513c3d5f-ad2d-4852-ad67-bdbfc750fa42-kube-api-access-w6kds\") pod \"maas-controller-64bbdb967c-t228l\" (UID: \"513c3d5f-ad2d-4852-ad67-bdbfc750fa42\") " pod="opendatahub/maas-controller-64bbdb967c-t228l"
Jul 24 18:54:50.166791 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:50.166674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/513c3d5f-ad2d-4852-ad67-bdbfc750fa42-webhook-cert\") pod \"maas-controller-64bbdb967c-t228l\" (UID: \"513c3d5f-ad2d-4852-ad67-bdbfc750fa42\") " pod="opendatahub/maas-controller-64bbdb967c-t228l"
Jul 24 18:54:50.168975 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:50.168957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/513c3d5f-ad2d-4852-ad67-bdbfc750fa42-webhook-cert\") pod \"maas-controller-64bbdb967c-t228l\" (UID: \"513c3d5f-ad2d-4852-ad67-bdbfc750fa42\") " pod="opendatahub/maas-controller-64bbdb967c-t228l"
Jul 24 18:54:50.175736 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:50.175715    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6kds\" (UniqueName: \"kubernetes.io/projected/513c3d5f-ad2d-4852-ad67-bdbfc750fa42-kube-api-access-w6kds\") pod \"maas-controller-64bbdb967c-t228l\" (UID: \"513c3d5f-ad2d-4852-ad67-bdbfc750fa42\") " pod="opendatahub/maas-controller-64bbdb967c-t228l"
Jul 24 18:54:50.222944 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:50.222918    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64bbdb967c-t228l"
Jul 24 18:54:50.350373 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:50.350348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64bbdb967c-t228l"]
Jul 24 18:54:50.351833 ip-10-0-128-111 kubenswrapper[2580]: W0724 18:54:50.351810    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513c3d5f_ad2d_4852_ad67_bdbfc750fa42.slice/crio-c61ca636cdbe0a6df53ef27928493f88a3512d63c95e3e3cfc15e9ecbc624ca1 WatchSource:0}: Error finding container c61ca636cdbe0a6df53ef27928493f88a3512d63c95e3e3cfc15e9ecbc624ca1: Status 404 returned error can't find the container with id c61ca636cdbe0a6df53ef27928493f88a3512d63c95e3e3cfc15e9ecbc624ca1
Jul 24 18:54:51.129255 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:51.129218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64bbdb967c-t228l" event={"ID":"513c3d5f-ad2d-4852-ad67-bdbfc750fa42","Type":"ContainerStarted","Data":"499e82b46af50d5ced88e561c2d2c2bf9b13c808b2aaad8094d176978312fe5f"}
Jul 24 18:54:51.129728 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:51.129275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64bbdb967c-t228l" event={"ID":"513c3d5f-ad2d-4852-ad67-bdbfc750fa42","Type":"ContainerStarted","Data":"c61ca636cdbe0a6df53ef27928493f88a3512d63c95e3e3cfc15e9ecbc624ca1"}
Jul 24 18:54:51.129728 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:51.129307    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64bbdb967c-t228l"
Jul 24 18:54:51.149147 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:54:51.149100    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-64bbdb967c-t228l" podStartSLOduration=1.812538964 podStartE2EDuration="2.149085893s" totalImagesPullingTime="336.546929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:54:51.146795777 +0000 UTC m=+890.522455084" watchObservedRunningTime="2026-07-24 18:54:51.149085893 +0000 UTC m=+890.524745190"
Jul 24 18:55:01.108039 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:55:01.107959    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 18:55:01.113804 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:55:01.113778    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 18:55:02.136619 ip-10-0-128-111 kubenswrapper[2580]: I0724 18:55:02.136586    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64bbdb967c-t228l"
Jul 24 19:00:01.139474 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:00:01.139435    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:00:01.141952 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:00:01.141916    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:05:01.169226 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:05:01.169197    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:05:01.169808 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:05:01.169655    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:07:55.031711 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.031682    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"]
Jul 24 19:07:55.035408 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.035391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:07:55.040308 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.040289    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1ed084fc-serving-cert\""
Jul 24 19:07:55.047263 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.047239    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"]
Jul 24 19:07:55.144438 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.144400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da740ae-7dc6-4528-8989-8881fe240f07-openshift-service-ca\") pod \"maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l\" (UID: \"5da740ae-7dc6-4528-8989-8881fe240f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:07:55.144597 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.144472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5da740ae-7dc6-4528-8989-8881fe240f07-maas-api-tls\") pod \"maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l\" (UID: \"5da740ae-7dc6-4528-8989-8881fe240f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:07:55.144597 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.144520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxlln\" (UniqueName: \"kubernetes.io/projected/5da740ae-7dc6-4528-8989-8881fe240f07-kube-api-access-qxlln\") pod \"maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l\" (UID: \"5da740ae-7dc6-4528-8989-8881fe240f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:07:55.245341 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.245306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da740ae-7dc6-4528-8989-8881fe240f07-openshift-service-ca\") pod \"maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l\" (UID: \"5da740ae-7dc6-4528-8989-8881fe240f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:07:55.245520 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.245356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5da740ae-7dc6-4528-8989-8881fe240f07-maas-api-tls\") pod \"maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l\" (UID: \"5da740ae-7dc6-4528-8989-8881fe240f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:07:55.245520 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.245410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxlln\" (UniqueName: \"kubernetes.io/projected/5da740ae-7dc6-4528-8989-8881fe240f07-kube-api-access-qxlln\") pod \"maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l\" (UID: \"5da740ae-7dc6-4528-8989-8881fe240f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:07:55.245988 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.245962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da740ae-7dc6-4528-8989-8881fe240f07-openshift-service-ca\") pod \"maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l\" (UID: \"5da740ae-7dc6-4528-8989-8881fe240f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:07:55.247860 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.247836    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5da740ae-7dc6-4528-8989-8881fe240f07-maas-api-tls\") pod \"maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l\" (UID: \"5da740ae-7dc6-4528-8989-8881fe240f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:07:55.253289 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.253270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxlln\" (UniqueName: \"kubernetes.io/projected/5da740ae-7dc6-4528-8989-8881fe240f07-kube-api-access-qxlln\") pod \"maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l\" (UID: \"5da740ae-7dc6-4528-8989-8881fe240f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:07:55.345311 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.345290    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:07:55.474866 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.474801    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"]
Jul 24 19:07:55.477104 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:07:55.477077    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da740ae_7dc6_4528_8989_8881fe240f07.slice/crio-76b621fe0dd4b7947edb4a59391e38a0c2b2f58c849986faf1066e918fdd0489 WatchSource:0}: Error finding container 76b621fe0dd4b7947edb4a59391e38a0c2b2f58c849986faf1066e918fdd0489: Status 404 returned error can't find the container with id 76b621fe0dd4b7947edb4a59391e38a0c2b2f58c849986faf1066e918fdd0489
Jul 24 19:07:55.478514 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.478492    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 19:07:55.729303 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.729219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l" event={"ID":"5da740ae-7dc6-4528-8989-8881fe240f07","Type":"ContainerStarted","Data":"76b621fe0dd4b7947edb4a59391e38a0c2b2f58c849986faf1066e918fdd0489"}
Jul 24 19:07:55.772361 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.772330    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"]
Jul 24 19:07:55.776999 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.776981    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"
Jul 24 19:07:55.779066 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.779050    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1ed084fc\""
Jul 24 19:07:55.779146 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.779092    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1ed084fc-dockercfg-xm5km\""
Jul 24 19:07:55.788264 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.788241    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"]
Jul 24 19:07:55.802369 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.802344    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"]
Jul 24 19:07:55.805803 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.805784    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"
Jul 24 19:07:55.813898 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.813877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"]
Jul 24 19:07:55.851596 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.851570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68d9840-454b-4554-8a10-158ca6a5c439-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8\" (UID: \"e68d9840-454b-4554-8a10-158ca6a5c439\") " pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"
Jul 24 19:07:55.851728 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.851638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sccg\" (UniqueName: \"kubernetes.io/projected/e68d9840-454b-4554-8a10-158ca6a5c439-kube-api-access-7sccg\") pod \"payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8\" (UID: \"e68d9840-454b-4554-8a10-158ca6a5c439\") " pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"
Jul 24 19:07:55.952519 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.952482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6zgk\" (UniqueName: \"kubernetes.io/projected/98fb0e43-86f1-4609-85cc-de186458eff3-kube-api-access-k6zgk\") pod \"payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67\" (UID: \"98fb0e43-86f1-4609-85cc-de186458eff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"
Jul 24 19:07:55.952658 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.952599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68d9840-454b-4554-8a10-158ca6a5c439-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8\" (UID: \"e68d9840-454b-4554-8a10-158ca6a5c439\") " pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"
Jul 24 19:07:55.952658 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.952646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98fb0e43-86f1-4609-85cc-de186458eff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67\" (UID: \"98fb0e43-86f1-4609-85cc-de186458eff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"
Jul 24 19:07:55.952744 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.952684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sccg\" (UniqueName: \"kubernetes.io/projected/e68d9840-454b-4554-8a10-158ca6a5c439-kube-api-access-7sccg\") pod \"payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8\" (UID: \"e68d9840-454b-4554-8a10-158ca6a5c439\") " pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"
Jul 24 19:07:55.953274 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.953254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68d9840-454b-4554-8a10-158ca6a5c439-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8\" (UID: \"e68d9840-454b-4554-8a10-158ca6a5c439\") " pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"
Jul 24 19:07:55.961197 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:55.961166    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sccg\" (UniqueName: \"kubernetes.io/projected/e68d9840-454b-4554-8a10-158ca6a5c439-kube-api-access-7sccg\") pod \"payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8\" (UID: \"e68d9840-454b-4554-8a10-158ca6a5c439\") " pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"
Jul 24 19:07:56.053371 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.053346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98fb0e43-86f1-4609-85cc-de186458eff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67\" (UID: \"98fb0e43-86f1-4609-85cc-de186458eff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"
Jul 24 19:07:56.053692 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.053415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6zgk\" (UniqueName: \"kubernetes.io/projected/98fb0e43-86f1-4609-85cc-de186458eff3-kube-api-access-k6zgk\") pod \"payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67\" (UID: \"98fb0e43-86f1-4609-85cc-de186458eff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"
Jul 24 19:07:56.054070 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.054045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98fb0e43-86f1-4609-85cc-de186458eff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67\" (UID: \"98fb0e43-86f1-4609-85cc-de186458eff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"
Jul 24 19:07:56.060188 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.060168    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6zgk\" (UniqueName: \"kubernetes.io/projected/98fb0e43-86f1-4609-85cc-de186458eff3-kube-api-access-k6zgk\") pod \"payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67\" (UID: \"98fb0e43-86f1-4609-85cc-de186458eff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"
Jul 24 19:07:56.088067 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.088039    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"
Jul 24 19:07:56.116991 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.116949    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"
Jul 24 19:07:56.263157 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.263129    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"]
Jul 24 19:07:56.265687 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:07:56.265660    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fb0e43_86f1_4609_85cc_de186458eff3.slice/crio-a598962caf3b5b64944635755ed0c0982bedcb07e651ea42c713afdc767e6648 WatchSource:0}: Error finding container a598962caf3b5b64944635755ed0c0982bedcb07e651ea42c713afdc767e6648: Status 404 returned error can't find the container with id a598962caf3b5b64944635755ed0c0982bedcb07e651ea42c713afdc767e6648
Jul 24 19:07:56.442430 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.442094    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"]
Jul 24 19:07:56.444964 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:07:56.444933    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68d9840_454b_4554_8a10_158ca6a5c439.slice/crio-ccd4324f6224424f426585af0314d3914d5d4bab0473b0d6d0fa164244ffca71 WatchSource:0}: Error finding container ccd4324f6224424f426585af0314d3914d5d4bab0473b0d6d0fa164244ffca71: Status 404 returned error can't find the container with id ccd4324f6224424f426585af0314d3914d5d4bab0473b0d6d0fa164244ffca71
Jul 24 19:07:56.736059 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.736013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l" event={"ID":"5da740ae-7dc6-4528-8989-8881fe240f07","Type":"ContainerStarted","Data":"f4d0e90cb4f5370b0795840cb8877c4850576bbd627be8ddd36e3c8f649d9890"}
Jul 24 19:07:56.736260 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.736092    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:07:56.737698 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.737647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67" event={"ID":"98fb0e43-86f1-4609-85cc-de186458eff3","Type":"ContainerStarted","Data":"d646b85e6abcd96db8be547db837c69200c1bca929895d7d0c17301a1025b436"}
Jul 24 19:07:56.737827 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.737699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67" event={"ID":"98fb0e43-86f1-4609-85cc-de186458eff3","Type":"ContainerStarted","Data":"a598962caf3b5b64944635755ed0c0982bedcb07e651ea42c713afdc767e6648"}
Jul 24 19:07:56.737827 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.737731    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"
Jul 24 19:07:56.739246 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.739218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8" event={"ID":"e68d9840-454b-4554-8a10-158ca6a5c439","Type":"ContainerStarted","Data":"0a80ff67d33eacff0a9404081254f86df01bceaa9bd5658f8bff92955b9c0107"}
Jul 24 19:07:56.739371 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.739266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8" event={"ID":"e68d9840-454b-4554-8a10-158ca6a5c439","Type":"ContainerStarted","Data":"ccd4324f6224424f426585af0314d3914d5d4bab0473b0d6d0fa164244ffca71"}
Jul 24 19:07:56.739371 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.739303    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"
Jul 24 19:07:56.761627 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.761584    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l" podStartSLOduration=1.249591106 podStartE2EDuration="1.76157025s" totalImagesPullingTime="511.979144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:07:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:07:56.759780125 +0000 UTC m=+1676.135439420" watchObservedRunningTime="2026-07-24 19:07:56.76157025 +0000 UTC m=+1676.137229546"
Jul 24 19:07:56.795924 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.795882    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67" podStartSLOduration=1.7958687709999999 podStartE2EDuration="1.795868771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:07:56.794170899 +0000 UTC m=+1676.169830199" watchObservedRunningTime="2026-07-24 19:07:56.795868771 +0000 UTC m=+1676.171528067"
Jul 24 19:07:56.824027 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:07:56.823985    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8" podStartSLOduration=1.823974247 podStartE2EDuration="1.823974247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:07:56.82151504 +0000 UTC m=+1676.197174336" watchObservedRunningTime="2026-07-24 19:07:56.823974247 +0000 UTC m=+1676.199633543"
Jul 24 19:08:02.750223 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:02.750195    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:08:07.745816 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:07.745784    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"
Jul 24 19:08:07.746265 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:07.745851    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"
Jul 24 19:08:30.879155 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:30.879080    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"]
Jul 24 19:08:30.879701 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:30.879311    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l" podUID="5da740ae-7dc6-4528-8989-8881fe240f07" containerName="maas-api" containerID="cri-o://f4d0e90cb4f5370b0795840cb8877c4850576bbd627be8ddd36e3c8f649d9890" gracePeriod=30
Jul 24 19:08:30.928077 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:30.928044    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"]
Jul 24 19:08:30.928300 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:30.928254    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8" podUID="e68d9840-454b-4554-8a10-158ca6a5c439" containerName="payload-processing" containerID="cri-o://0a80ff67d33eacff0a9404081254f86df01bceaa9bd5658f8bff92955b9c0107" gracePeriod=30
Jul 24 19:08:30.935017 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:30.934982    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"]
Jul 24 19:08:30.935197 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:30.935175    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67" podUID="98fb0e43-86f1-4609-85cc-de186458eff3" containerName="payload-pre-processing" containerID="cri-o://d646b85e6abcd96db8be547db837c69200c1bca929895d7d0c17301a1025b436" gracePeriod=30
Jul 24 19:08:31.205222 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.205185    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"
Jul 24 19:08:31.208826 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.208808    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:08:31.234313 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.234290    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"
Jul 24 19:08:31.245458 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.245438    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5da740ae-7dc6-4528-8989-8881fe240f07-maas-api-tls\") pod \"5da740ae-7dc6-4528-8989-8881fe240f07\" (UID: \"5da740ae-7dc6-4528-8989-8881fe240f07\") "
Jul 24 19:08:31.245577 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.245477    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxlln\" (UniqueName: \"kubernetes.io/projected/5da740ae-7dc6-4528-8989-8881fe240f07-kube-api-access-qxlln\") pod \"5da740ae-7dc6-4528-8989-8881fe240f07\" (UID: \"5da740ae-7dc6-4528-8989-8881fe240f07\") "
Jul 24 19:08:31.245577 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.245509    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sccg\" (UniqueName: \"kubernetes.io/projected/e68d9840-454b-4554-8a10-158ca6a5c439-kube-api-access-7sccg\") pod \"e68d9840-454b-4554-8a10-158ca6a5c439\" (UID: \"e68d9840-454b-4554-8a10-158ca6a5c439\") "
Jul 24 19:08:31.245577 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.245529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da740ae-7dc6-4528-8989-8881fe240f07-openshift-service-ca\") pod \"5da740ae-7dc6-4528-8989-8881fe240f07\" (UID: \"5da740ae-7dc6-4528-8989-8881fe240f07\") "
Jul 24 19:08:31.245747 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.245590    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68d9840-454b-4554-8a10-158ca6a5c439-plugins-config-volume\") pod \"e68d9840-454b-4554-8a10-158ca6a5c439\" (UID: \"e68d9840-454b-4554-8a10-158ca6a5c439\") "
Jul 24 19:08:31.246204 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.246174    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68d9840-454b-4554-8a10-158ca6a5c439-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e68d9840-454b-4554-8a10-158ca6a5c439" (UID: "e68d9840-454b-4554-8a10-158ca6a5c439"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:08:31.246310 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.246216    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da740ae-7dc6-4528-8989-8881fe240f07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5da740ae-7dc6-4528-8989-8881fe240f07" (UID: "5da740ae-7dc6-4528-8989-8881fe240f07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:08:31.247763 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.247727    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da740ae-7dc6-4528-8989-8881fe240f07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5da740ae-7dc6-4528-8989-8881fe240f07" (UID: "5da740ae-7dc6-4528-8989-8881fe240f07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:08:31.247849 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.247775    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da740ae-7dc6-4528-8989-8881fe240f07-kube-api-access-qxlln" (OuterVolumeSpecName: "kube-api-access-qxlln") pod "5da740ae-7dc6-4528-8989-8881fe240f07" (UID: "5da740ae-7dc6-4528-8989-8881fe240f07"). InnerVolumeSpecName "kube-api-access-qxlln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:08:31.247888 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.247848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68d9840-454b-4554-8a10-158ca6a5c439-kube-api-access-7sccg" (OuterVolumeSpecName: "kube-api-access-7sccg") pod "e68d9840-454b-4554-8a10-158ca6a5c439" (UID: "e68d9840-454b-4554-8a10-158ca6a5c439"). InnerVolumeSpecName "kube-api-access-7sccg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:08:31.347139 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.347102    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98fb0e43-86f1-4609-85cc-de186458eff3-plugins-config-volume\") pod \"98fb0e43-86f1-4609-85cc-de186458eff3\" (UID: \"98fb0e43-86f1-4609-85cc-de186458eff3\") "
Jul 24 19:08:31.347327 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.347163    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6zgk\" (UniqueName: \"kubernetes.io/projected/98fb0e43-86f1-4609-85cc-de186458eff3-kube-api-access-k6zgk\") pod \"98fb0e43-86f1-4609-85cc-de186458eff3\" (UID: \"98fb0e43-86f1-4609-85cc-de186458eff3\") "
Jul 24 19:08:31.347327 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.347308    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5da740ae-7dc6-4528-8989-8881fe240f07-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:08:31.347327 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.347318    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxlln\" (UniqueName: \"kubernetes.io/projected/5da740ae-7dc6-4528-8989-8881fe240f07-kube-api-access-qxlln\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:08:31.347327 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.347329    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sccg\" (UniqueName: \"kubernetes.io/projected/e68d9840-454b-4554-8a10-158ca6a5c439-kube-api-access-7sccg\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:08:31.347574 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.347338    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da740ae-7dc6-4528-8989-8881fe240f07-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:08:31.347574 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.347347    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68d9840-454b-4554-8a10-158ca6a5c439-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:08:31.347574 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.347480    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fb0e43-86f1-4609-85cc-de186458eff3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98fb0e43-86f1-4609-85cc-de186458eff3" (UID: "98fb0e43-86f1-4609-85cc-de186458eff3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:08:31.349179 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.349155    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fb0e43-86f1-4609-85cc-de186458eff3-kube-api-access-k6zgk" (OuterVolumeSpecName: "kube-api-access-k6zgk") pod "98fb0e43-86f1-4609-85cc-de186458eff3" (UID: "98fb0e43-86f1-4609-85cc-de186458eff3"). InnerVolumeSpecName "kube-api-access-k6zgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:08:31.448478 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.448416    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98fb0e43-86f1-4609-85cc-de186458eff3-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:08:31.448478 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.448437    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6zgk\" (UniqueName: \"kubernetes.io/projected/98fb0e43-86f1-4609-85cc-de186458eff3-kube-api-access-k6zgk\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:08:31.907208 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.907178    2580 generic.go:356] "Generic (PLEG): container finished" podID="98fb0e43-86f1-4609-85cc-de186458eff3" containerID="d646b85e6abcd96db8be547db837c69200c1bca929895d7d0c17301a1025b436" exitCode=0
Jul 24 19:08:31.907577 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.907219    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"
Jul 24 19:08:31.907577 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.907263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67" event={"ID":"98fb0e43-86f1-4609-85cc-de186458eff3","Type":"ContainerDied","Data":"d646b85e6abcd96db8be547db837c69200c1bca929895d7d0c17301a1025b436"}
Jul 24 19:08:31.907577 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.907312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67" event={"ID":"98fb0e43-86f1-4609-85cc-de186458eff3","Type":"ContainerDied","Data":"a598962caf3b5b64944635755ed0c0982bedcb07e651ea42c713afdc767e6648"}
Jul 24 19:08:31.907577 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.907339    2580 scope.go:117] "RemoveContainer" containerID="d646b85e6abcd96db8be547db837c69200c1bca929895d7d0c17301a1025b436"
Jul 24 19:08:31.908678 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.908619    2580 generic.go:356] "Generic (PLEG): container finished" podID="e68d9840-454b-4554-8a10-158ca6a5c439" containerID="0a80ff67d33eacff0a9404081254f86df01bceaa9bd5658f8bff92955b9c0107" exitCode=0
Jul 24 19:08:31.908678 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.908668    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"
Jul 24 19:08:31.908792 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.908693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8" event={"ID":"e68d9840-454b-4554-8a10-158ca6a5c439","Type":"ContainerDied","Data":"0a80ff67d33eacff0a9404081254f86df01bceaa9bd5658f8bff92955b9c0107"}
Jul 24 19:08:31.908792 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.908744    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8" event={"ID":"e68d9840-454b-4554-8a10-158ca6a5c439","Type":"ContainerDied","Data":"ccd4324f6224424f426585af0314d3914d5d4bab0473b0d6d0fa164244ffca71"}
Jul 24 19:08:31.910048 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.910027    2580 generic.go:356] "Generic (PLEG): container finished" podID="5da740ae-7dc6-4528-8989-8881fe240f07" containerID="f4d0e90cb4f5370b0795840cb8877c4850576bbd627be8ddd36e3c8f649d9890" exitCode=0
Jul 24 19:08:31.910158 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.910053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l" event={"ID":"5da740ae-7dc6-4528-8989-8881fe240f07","Type":"ContainerDied","Data":"f4d0e90cb4f5370b0795840cb8877c4850576bbd627be8ddd36e3c8f649d9890"}
Jul 24 19:08:31.910158 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.910066    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"
Jul 24 19:08:31.910158 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.910082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l" event={"ID":"5da740ae-7dc6-4528-8989-8881fe240f07","Type":"ContainerDied","Data":"76b621fe0dd4b7947edb4a59391e38a0c2b2f58c849986faf1066e918fdd0489"}
Jul 24 19:08:31.917036 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.917014    2580 scope.go:117] "RemoveContainer" containerID="d646b85e6abcd96db8be547db837c69200c1bca929895d7d0c17301a1025b436"
Jul 24 19:08:31.917293 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:08:31.917274    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d646b85e6abcd96db8be547db837c69200c1bca929895d7d0c17301a1025b436\": container with ID starting with d646b85e6abcd96db8be547db837c69200c1bca929895d7d0c17301a1025b436 not found: ID does not exist" containerID="d646b85e6abcd96db8be547db837c69200c1bca929895d7d0c17301a1025b436"
Jul 24 19:08:31.917363 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.917300    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d646b85e6abcd96db8be547db837c69200c1bca929895d7d0c17301a1025b436"} err="failed to get container status \"d646b85e6abcd96db8be547db837c69200c1bca929895d7d0c17301a1025b436\": rpc error: code = NotFound desc = could not find container \"d646b85e6abcd96db8be547db837c69200c1bca929895d7d0c17301a1025b436\": container with ID starting with d646b85e6abcd96db8be547db837c69200c1bca929895d7d0c17301a1025b436 not found: ID does not exist"
Jul 24 19:08:31.917363 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.917316    2580 scope.go:117] "RemoveContainer" containerID="0a80ff67d33eacff0a9404081254f86df01bceaa9bd5658f8bff92955b9c0107"
Jul 24 19:08:31.925699 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.925682    2580 scope.go:117] "RemoveContainer" containerID="0a80ff67d33eacff0a9404081254f86df01bceaa9bd5658f8bff92955b9c0107"
Jul 24 19:08:31.925941 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:08:31.925916    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a80ff67d33eacff0a9404081254f86df01bceaa9bd5658f8bff92955b9c0107\": container with ID starting with 0a80ff67d33eacff0a9404081254f86df01bceaa9bd5658f8bff92955b9c0107 not found: ID does not exist" containerID="0a80ff67d33eacff0a9404081254f86df01bceaa9bd5658f8bff92955b9c0107"
Jul 24 19:08:31.925998 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.925950    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a80ff67d33eacff0a9404081254f86df01bceaa9bd5658f8bff92955b9c0107"} err="failed to get container status \"0a80ff67d33eacff0a9404081254f86df01bceaa9bd5658f8bff92955b9c0107\": rpc error: code = NotFound desc = could not find container \"0a80ff67d33eacff0a9404081254f86df01bceaa9bd5658f8bff92955b9c0107\": container with ID starting with 0a80ff67d33eacff0a9404081254f86df01bceaa9bd5658f8bff92955b9c0107 not found: ID does not exist"
Jul 24 19:08:31.925998 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.925970    2580 scope.go:117] "RemoveContainer" containerID="f4d0e90cb4f5370b0795840cb8877c4850576bbd627be8ddd36e3c8f649d9890"
Jul 24 19:08:31.933681 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.933665    2580 scope.go:117] "RemoveContainer" containerID="f4d0e90cb4f5370b0795840cb8877c4850576bbd627be8ddd36e3c8f649d9890"
Jul 24 19:08:31.933932 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:08:31.933914    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d0e90cb4f5370b0795840cb8877c4850576bbd627be8ddd36e3c8f649d9890\": container with ID starting with f4d0e90cb4f5370b0795840cb8877c4850576bbd627be8ddd36e3c8f649d9890 not found: ID does not exist" containerID="f4d0e90cb4f5370b0795840cb8877c4850576bbd627be8ddd36e3c8f649d9890"
Jul 24 19:08:31.933988 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.933928    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"]
Jul 24 19:08:31.933988 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.933937    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d0e90cb4f5370b0795840cb8877c4850576bbd627be8ddd36e3c8f649d9890"} err="failed to get container status \"f4d0e90cb4f5370b0795840cb8877c4850576bbd627be8ddd36e3c8f649d9890\": rpc error: code = NotFound desc = could not find container \"f4d0e90cb4f5370b0795840cb8877c4850576bbd627be8ddd36e3c8f649d9890\": container with ID starting with f4d0e90cb4f5370b0795840cb8877c4850576bbd627be8ddd36e3c8f649d9890 not found: ID does not exist"
Jul 24 19:08:31.940062 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.939949    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ed084fc-7bd74bc8bd-jhz67"]
Jul 24 19:08:31.952748 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.952725    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"]
Jul 24 19:08:31.956439 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.956417    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ed084fc-79ff79b6cd-xt29l"]
Jul 24 19:08:31.965372 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.965352    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"]
Jul 24 19:08:31.968038 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:31.968011    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ed084fc-57855b7654-zhtv8"]
Jul 24 19:08:33.197516 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:33.197476    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5da740ae-7dc6-4528-8989-8881fe240f07" path="/var/lib/kubelet/pods/5da740ae-7dc6-4528-8989-8881fe240f07/volumes"
Jul 24 19:08:33.198013 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:33.197970    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98fb0e43-86f1-4609-85cc-de186458eff3" path="/var/lib/kubelet/pods/98fb0e43-86f1-4609-85cc-de186458eff3/volumes"
Jul 24 19:08:33.198350 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:33.198330    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e68d9840-454b-4554-8a10-158ca6a5c439" path="/var/lib/kubelet/pods/e68d9840-454b-4554-8a10-158ca6a5c439/volumes"
Jul 24 19:08:42.295376 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.295342    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"]
Jul 24 19:08:42.295759 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.295740    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da740ae-7dc6-4528-8989-8881fe240f07" containerName="maas-api"
Jul 24 19:08:42.295759 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.295752    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da740ae-7dc6-4528-8989-8881fe240f07" containerName="maas-api"
Jul 24 19:08:42.295826 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.295771    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68d9840-454b-4554-8a10-158ca6a5c439" containerName="payload-processing"
Jul 24 19:08:42.295826 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.295780    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68d9840-454b-4554-8a10-158ca6a5c439" containerName="payload-processing"
Jul 24 19:08:42.295826 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.295795    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98fb0e43-86f1-4609-85cc-de186458eff3" containerName="payload-pre-processing"
Jul 24 19:08:42.295826 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.295801    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98fb0e43-86f1-4609-85cc-de186458eff3" containerName="payload-pre-processing"
Jul 24 19:08:42.295945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.295866    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="98fb0e43-86f1-4609-85cc-de186458eff3" containerName="payload-pre-processing"
Jul 24 19:08:42.295945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.295877    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da740ae-7dc6-4528-8989-8881fe240f07" containerName="maas-api"
Jul 24 19:08:42.295945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.295884    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68d9840-454b-4554-8a10-158ca6a5c439" containerName="payload-processing"
Jul 24 19:08:42.300102 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.300084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:08:42.302583 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.302563    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-889a9e38-serving-cert\""
Jul 24 19:08:42.320342 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.320319    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"]
Jul 24 19:08:42.443752 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.443718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqwmz\" (UniqueName: \"kubernetes.io/projected/c7f097a9-49d1-458f-8f26-ee2c929874a8-kube-api-access-vqwmz\") pod \"maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd\" (UID: \"c7f097a9-49d1-458f-8f26-ee2c929874a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:08:42.443926 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.443773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f097a9-49d1-458f-8f26-ee2c929874a8-openshift-service-ca\") pod \"maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd\" (UID: \"c7f097a9-49d1-458f-8f26-ee2c929874a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:08:42.443926 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.443857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7f097a9-49d1-458f-8f26-ee2c929874a8-maas-api-tls\") pod \"maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd\" (UID: \"c7f097a9-49d1-458f-8f26-ee2c929874a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:08:42.545101 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.545064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f097a9-49d1-458f-8f26-ee2c929874a8-openshift-service-ca\") pod \"maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd\" (UID: \"c7f097a9-49d1-458f-8f26-ee2c929874a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:08:42.545258 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.545128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7f097a9-49d1-458f-8f26-ee2c929874a8-maas-api-tls\") pod \"maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd\" (UID: \"c7f097a9-49d1-458f-8f26-ee2c929874a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:08:42.545258 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.545198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqwmz\" (UniqueName: \"kubernetes.io/projected/c7f097a9-49d1-458f-8f26-ee2c929874a8-kube-api-access-vqwmz\") pod \"maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd\" (UID: \"c7f097a9-49d1-458f-8f26-ee2c929874a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:08:42.545258 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:08:42.545219    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-serving-cert: secret "maas-api-e2e-migrate-889a9e38-serving-cert" not found
Jul 24 19:08:42.545362 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:08:42.545294    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f097a9-49d1-458f-8f26-ee2c929874a8-maas-api-tls podName:c7f097a9-49d1-458f-8f26-ee2c929874a8 nodeName:}" failed. No retries permitted until 2026-07-24 19:08:43.045279083 +0000 UTC m=+1722.420938361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7f097a9-49d1-458f-8f26-ee2c929874a8-maas-api-tls") pod "maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd" (UID: "c7f097a9-49d1-458f-8f26-ee2c929874a8") : secret "maas-api-e2e-migrate-889a9e38-serving-cert" not found
Jul 24 19:08:42.545825 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.545782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f097a9-49d1-458f-8f26-ee2c929874a8-openshift-service-ca\") pod \"maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd\" (UID: \"c7f097a9-49d1-458f-8f26-ee2c929874a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:08:42.554754 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.554728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqwmz\" (UniqueName: \"kubernetes.io/projected/c7f097a9-49d1-458f-8f26-ee2c929874a8-kube-api-access-vqwmz\") pod \"maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd\" (UID: \"c7f097a9-49d1-458f-8f26-ee2c929874a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:08:42.952970 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.952935    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"]
Jul 24 19:08:42.956498 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.956483    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"
Jul 24 19:08:42.958992 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.958967    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-889a9e38\""
Jul 24 19:08:42.959125 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.959021    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-889a9e38-dockercfg-qss7m\""
Jul 24 19:08:42.965127 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.965106    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"]
Jul 24 19:08:42.994817 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.994790    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"]
Jul 24 19:08:42.998511 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:42.998492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"
Jul 24 19:08:43.012871 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.012843    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"]
Jul 24 19:08:43.050610 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.050577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfjnf\" (UniqueName: \"kubernetes.io/projected/5602c815-e107-4f98-b6e3-780f4beed257-kube-api-access-jfjnf\") pod \"payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp\" (UID: \"5602c815-e107-4f98-b6e3-780f4beed257\") " pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"
Jul 24 19:08:43.050767 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.050654    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ff79d-5f8f-4bf0-96c3-3dc489e99572-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr\" (UID: \"7e1ff79d-5f8f-4bf0-96c3-3dc489e99572\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"
Jul 24 19:08:43.050767 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.050708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7f097a9-49d1-458f-8f26-ee2c929874a8-maas-api-tls\") pod \"maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd\" (UID: \"c7f097a9-49d1-458f-8f26-ee2c929874a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:08:43.050849 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.050771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zcpg\" (UniqueName: \"kubernetes.io/projected/7e1ff79d-5f8f-4bf0-96c3-3dc489e99572-kube-api-access-2zcpg\") pod \"payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr\" (UID: \"7e1ff79d-5f8f-4bf0-96c3-3dc489e99572\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"
Jul 24 19:08:43.050849 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.050791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5602c815-e107-4f98-b6e3-780f4beed257-plugins-config-volume\") pod \"payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp\" (UID: \"5602c815-e107-4f98-b6e3-780f4beed257\") " pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"
Jul 24 19:08:43.053288 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.053256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7f097a9-49d1-458f-8f26-ee2c929874a8-maas-api-tls\") pod \"maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd\" (UID: \"c7f097a9-49d1-458f-8f26-ee2c929874a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:08:43.151607 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.151568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ff79d-5f8f-4bf0-96c3-3dc489e99572-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr\" (UID: \"7e1ff79d-5f8f-4bf0-96c3-3dc489e99572\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"
Jul 24 19:08:43.151763 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.151674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zcpg\" (UniqueName: \"kubernetes.io/projected/7e1ff79d-5f8f-4bf0-96c3-3dc489e99572-kube-api-access-2zcpg\") pod \"payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr\" (UID: \"7e1ff79d-5f8f-4bf0-96c3-3dc489e99572\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"
Jul 24 19:08:43.151763 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.151703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5602c815-e107-4f98-b6e3-780f4beed257-plugins-config-volume\") pod \"payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp\" (UID: \"5602c815-e107-4f98-b6e3-780f4beed257\") " pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"
Jul 24 19:08:43.151763 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.151747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjnf\" (UniqueName: \"kubernetes.io/projected/5602c815-e107-4f98-b6e3-780f4beed257-kube-api-access-jfjnf\") pod \"payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp\" (UID: \"5602c815-e107-4f98-b6e3-780f4beed257\") " pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"
Jul 24 19:08:43.152353 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.152330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ff79d-5f8f-4bf0-96c3-3dc489e99572-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr\" (UID: \"7e1ff79d-5f8f-4bf0-96c3-3dc489e99572\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"
Jul 24 19:08:43.152517 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.152496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5602c815-e107-4f98-b6e3-780f4beed257-plugins-config-volume\") pod \"payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp\" (UID: \"5602c815-e107-4f98-b6e3-780f4beed257\") " pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"
Jul 24 19:08:43.159283 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.159255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zcpg\" (UniqueName: \"kubernetes.io/projected/7e1ff79d-5f8f-4bf0-96c3-3dc489e99572-kube-api-access-2zcpg\") pod \"payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr\" (UID: \"7e1ff79d-5f8f-4bf0-96c3-3dc489e99572\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"
Jul 24 19:08:43.159367 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.159302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfjnf\" (UniqueName: \"kubernetes.io/projected/5602c815-e107-4f98-b6e3-780f4beed257-kube-api-access-jfjnf\") pod \"payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp\" (UID: \"5602c815-e107-4f98-b6e3-780f4beed257\") " pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"
Jul 24 19:08:43.212080 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.212005    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:08:43.268907 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.268879    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"
Jul 24 19:08:43.307840 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.307806    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"
Jul 24 19:08:43.425909 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.425881    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"]
Jul 24 19:08:43.430093 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:08:43.430036    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5602c815_e107_4f98_b6e3_780f4beed257.slice/crio-20041f64e9ab987e1c60a33edb7b21287ca9852c032f41c50acc138eeca333ad WatchSource:0}: Error finding container 20041f64e9ab987e1c60a33edb7b21287ca9852c032f41c50acc138eeca333ad: Status 404 returned error can't find the container with id 20041f64e9ab987e1c60a33edb7b21287ca9852c032f41c50acc138eeca333ad
Jul 24 19:08:43.555077 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.555053    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"]
Jul 24 19:08:43.557824 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:08:43.557789    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f097a9_49d1_458f_8f26_ee2c929874a8.slice/crio-959aadbfb3bd80c734ff3fb890d436bbf255946640ca912b8d603538a14f611f WatchSource:0}: Error finding container 959aadbfb3bd80c734ff3fb890d436bbf255946640ca912b8d603538a14f611f: Status 404 returned error can't find the container with id 959aadbfb3bd80c734ff3fb890d436bbf255946640ca912b8d603538a14f611f
Jul 24 19:08:43.663518 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.663491    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"]
Jul 24 19:08:43.665887 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:08:43.665858    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1ff79d_5f8f_4bf0_96c3_3dc489e99572.slice/crio-ee5b2d62da6fd75f9ee618574d79af729e9262b6c7d69d1b10e29094d384b7c0 WatchSource:0}: Error finding container ee5b2d62da6fd75f9ee618574d79af729e9262b6c7d69d1b10e29094d384b7c0: Status 404 returned error can't find the container with id ee5b2d62da6fd75f9ee618574d79af729e9262b6c7d69d1b10e29094d384b7c0
Jul 24 19:08:43.970678 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.967805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr" event={"ID":"7e1ff79d-5f8f-4bf0-96c3-3dc489e99572","Type":"ContainerStarted","Data":"830784dba32a5c05b11ec963e0011761352ea092b60f153b2dbd83acd5e5f975"}
Jul 24 19:08:43.970678 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.967868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr" event={"ID":"7e1ff79d-5f8f-4bf0-96c3-3dc489e99572","Type":"ContainerStarted","Data":"ee5b2d62da6fd75f9ee618574d79af729e9262b6c7d69d1b10e29094d384b7c0"}
Jul 24 19:08:43.970678 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.968059    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"
Jul 24 19:08:43.972862 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.972834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp" event={"ID":"5602c815-e107-4f98-b6e3-780f4beed257","Type":"ContainerStarted","Data":"5548105fb6a7c420ae3a9129168fab3a3098bb21b679fae6beffb1618ad95de7"}
Jul 24 19:08:43.972972 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.972883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp" event={"ID":"5602c815-e107-4f98-b6e3-780f4beed257","Type":"ContainerStarted","Data":"20041f64e9ab987e1c60a33edb7b21287ca9852c032f41c50acc138eeca333ad"}
Jul 24 19:08:43.972972 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.972916    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"
Jul 24 19:08:43.974107 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.974087    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd" event={"ID":"c7f097a9-49d1-458f-8f26-ee2c929874a8","Type":"ContainerStarted","Data":"959aadbfb3bd80c734ff3fb890d436bbf255946640ca912b8d603538a14f611f"}
Jul 24 19:08:43.984502 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.984464    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr" podStartSLOduration=1.984451416 podStartE2EDuration="1.984451416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:08:43.983467651 +0000 UTC m=+1723.359126949" watchObservedRunningTime="2026-07-24 19:08:43.984451416 +0000 UTC m=+1723.360110711"
Jul 24 19:08:43.998719 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:43.998679    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp" podStartSLOduration=1.998665905 podStartE2EDuration="1.998665905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:08:43.997127292 +0000 UTC m=+1723.372786600" watchObservedRunningTime="2026-07-24 19:08:43.998665905 +0000 UTC m=+1723.374325201"
Jul 24 19:08:44.980230 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:44.980186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd" event={"ID":"c7f097a9-49d1-458f-8f26-ee2c929874a8","Type":"ContainerStarted","Data":"00f0393bdc1683791f735999f37801c8576cb9e2592c0d2a8d52c1fccb0e7569"}
Jul 24 19:08:44.997761 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:44.997707    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd" podStartSLOduration=2.6225034750000003 podStartE2EDuration="2.997691644s" totalImagesPullingTime="375.188169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:08:44.995534899 +0000 UTC m=+1724.371194195" watchObservedRunningTime="2026-07-24 19:08:44.997691644 +0000 UTC m=+1724.373350939"
Jul 24 19:08:45.984268 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:45.984240    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:08:51.994737 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:51.994706    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:08:54.981971 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:54.981942    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"
Jul 24 19:08:54.982458 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:08:54.982022    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"
Jul 24 19:09:17.618222 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:17.618175    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"]
Jul 24 19:09:17.618781 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:17.618440    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd" podUID="c7f097a9-49d1-458f-8f26-ee2c929874a8" containerName="maas-api" containerID="cri-o://00f0393bdc1683791f735999f37801c8576cb9e2592c0d2a8d52c1fccb0e7569" gracePeriod=30
Jul 24 19:09:17.671068 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:17.670973    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"]
Jul 24 19:09:17.671525 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:17.671219    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp" podUID="5602c815-e107-4f98-b6e3-780f4beed257" containerName="payload-processing" containerID="cri-o://5548105fb6a7c420ae3a9129168fab3a3098bb21b679fae6beffb1618ad95de7" gracePeriod=30
Jul 24 19:09:17.676842 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:17.676804    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"]
Jul 24 19:09:17.677038 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:17.677010    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr" podUID="7e1ff79d-5f8f-4bf0-96c3-3dc489e99572" containerName="payload-pre-processing" containerID="cri-o://830784dba32a5c05b11ec963e0011761352ea092b60f153b2dbd83acd5e5f975" gracePeriod=30
Jul 24 19:09:17.948224 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:17.948200    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"
Jul 24 19:09:17.952121 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:17.952103    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:09:17.973071 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:17.973049    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"
Jul 24 19:09:18.043249 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.043217    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqwmz\" (UniqueName: \"kubernetes.io/projected/c7f097a9-49d1-458f-8f26-ee2c929874a8-kube-api-access-vqwmz\") pod \"c7f097a9-49d1-458f-8f26-ee2c929874a8\" (UID: \"c7f097a9-49d1-458f-8f26-ee2c929874a8\") "
Jul 24 19:09:18.043249 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.043253    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5602c815-e107-4f98-b6e3-780f4beed257-plugins-config-volume\") pod \"5602c815-e107-4f98-b6e3-780f4beed257\" (UID: \"5602c815-e107-4f98-b6e3-780f4beed257\") "
Jul 24 19:09:18.043459 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.043271    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zcpg\" (UniqueName: \"kubernetes.io/projected/7e1ff79d-5f8f-4bf0-96c3-3dc489e99572-kube-api-access-2zcpg\") pod \"7e1ff79d-5f8f-4bf0-96c3-3dc489e99572\" (UID: \"7e1ff79d-5f8f-4bf0-96c3-3dc489e99572\") "
Jul 24 19:09:18.043459 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.043316    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7f097a9-49d1-458f-8f26-ee2c929874a8-maas-api-tls\") pod \"c7f097a9-49d1-458f-8f26-ee2c929874a8\" (UID: \"c7f097a9-49d1-458f-8f26-ee2c929874a8\") "
Jul 24 19:09:18.043528 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.043464    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f097a9-49d1-458f-8f26-ee2c929874a8-openshift-service-ca\") pod \"c7f097a9-49d1-458f-8f26-ee2c929874a8\" (UID: \"c7f097a9-49d1-458f-8f26-ee2c929874a8\") "
Jul 24 19:09:18.043575 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.043548    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfjnf\" (UniqueName: \"kubernetes.io/projected/5602c815-e107-4f98-b6e3-780f4beed257-kube-api-access-jfjnf\") pod \"5602c815-e107-4f98-b6e3-780f4beed257\" (UID: \"5602c815-e107-4f98-b6e3-780f4beed257\") "
Jul 24 19:09:18.043629 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.043577    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ff79d-5f8f-4bf0-96c3-3dc489e99572-plugins-config-volume\") pod \"7e1ff79d-5f8f-4bf0-96c3-3dc489e99572\" (UID: \"7e1ff79d-5f8f-4bf0-96c3-3dc489e99572\") "
Jul 24 19:09:18.043937 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.043884    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5602c815-e107-4f98-b6e3-780f4beed257-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5602c815-e107-4f98-b6e3-780f4beed257" (UID: "5602c815-e107-4f98-b6e3-780f4beed257"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:09:18.043937 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.043907    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f097a9-49d1-458f-8f26-ee2c929874a8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7f097a9-49d1-458f-8f26-ee2c929874a8" (UID: "c7f097a9-49d1-458f-8f26-ee2c929874a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:09:18.044143 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.043973    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1ff79d-5f8f-4bf0-96c3-3dc489e99572-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e1ff79d-5f8f-4bf0-96c3-3dc489e99572" (UID: "7e1ff79d-5f8f-4bf0-96c3-3dc489e99572"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:09:18.045478 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.045458    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f097a9-49d1-458f-8f26-ee2c929874a8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7f097a9-49d1-458f-8f26-ee2c929874a8" (UID: "c7f097a9-49d1-458f-8f26-ee2c929874a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:09:18.045598 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.045584    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1ff79d-5f8f-4bf0-96c3-3dc489e99572-kube-api-access-2zcpg" (OuterVolumeSpecName: "kube-api-access-2zcpg") pod "7e1ff79d-5f8f-4bf0-96c3-3dc489e99572" (UID: "7e1ff79d-5f8f-4bf0-96c3-3dc489e99572"). InnerVolumeSpecName "kube-api-access-2zcpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:09:18.045830 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.045815    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f097a9-49d1-458f-8f26-ee2c929874a8-kube-api-access-vqwmz" (OuterVolumeSpecName: "kube-api-access-vqwmz") pod "c7f097a9-49d1-458f-8f26-ee2c929874a8" (UID: "c7f097a9-49d1-458f-8f26-ee2c929874a8"). InnerVolumeSpecName "kube-api-access-vqwmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:09:18.045867 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.045829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5602c815-e107-4f98-b6e3-780f4beed257-kube-api-access-jfjnf" (OuterVolumeSpecName: "kube-api-access-jfjnf") pod "5602c815-e107-4f98-b6e3-780f4beed257" (UID: "5602c815-e107-4f98-b6e3-780f4beed257"). InnerVolumeSpecName "kube-api-access-jfjnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:09:18.138371 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.138293    2580 generic.go:356] "Generic (PLEG): container finished" podID="5602c815-e107-4f98-b6e3-780f4beed257" containerID="5548105fb6a7c420ae3a9129168fab3a3098bb21b679fae6beffb1618ad95de7" exitCode=0
Jul 24 19:09:18.138371 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.138336    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"
Jul 24 19:09:18.138603 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.138393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp" event={"ID":"5602c815-e107-4f98-b6e3-780f4beed257","Type":"ContainerDied","Data":"5548105fb6a7c420ae3a9129168fab3a3098bb21b679fae6beffb1618ad95de7"}
Jul 24 19:09:18.138603 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.138450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp" event={"ID":"5602c815-e107-4f98-b6e3-780f4beed257","Type":"ContainerDied","Data":"20041f64e9ab987e1c60a33edb7b21287ca9852c032f41c50acc138eeca333ad"}
Jul 24 19:09:18.138603 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.138480    2580 scope.go:117] "RemoveContainer" containerID="5548105fb6a7c420ae3a9129168fab3a3098bb21b679fae6beffb1618ad95de7"
Jul 24 19:09:18.139695 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.139667    2580 generic.go:356] "Generic (PLEG): container finished" podID="c7f097a9-49d1-458f-8f26-ee2c929874a8" containerID="00f0393bdc1683791f735999f37801c8576cb9e2592c0d2a8d52c1fccb0e7569" exitCode=0
Jul 24 19:09:18.139801 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.139710    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"
Jul 24 19:09:18.139801 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.139709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd" event={"ID":"c7f097a9-49d1-458f-8f26-ee2c929874a8","Type":"ContainerDied","Data":"00f0393bdc1683791f735999f37801c8576cb9e2592c0d2a8d52c1fccb0e7569"}
Jul 24 19:09:18.139919 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.139817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd" event={"ID":"c7f097a9-49d1-458f-8f26-ee2c929874a8","Type":"ContainerDied","Data":"959aadbfb3bd80c734ff3fb890d436bbf255946640ca912b8d603538a14f611f"}
Jul 24 19:09:18.141109 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.141086    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e1ff79d-5f8f-4bf0-96c3-3dc489e99572" containerID="830784dba32a5c05b11ec963e0011761352ea092b60f153b2dbd83acd5e5f975" exitCode=0
Jul 24 19:09:18.141217 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.141126    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"
Jul 24 19:09:18.141217 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.141134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr" event={"ID":"7e1ff79d-5f8f-4bf0-96c3-3dc489e99572","Type":"ContainerDied","Data":"830784dba32a5c05b11ec963e0011761352ea092b60f153b2dbd83acd5e5f975"}
Jul 24 19:09:18.141217 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.141172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr" event={"ID":"7e1ff79d-5f8f-4bf0-96c3-3dc489e99572","Type":"ContainerDied","Data":"ee5b2d62da6fd75f9ee618574d79af729e9262b6c7d69d1b10e29094d384b7c0"}
Jul 24 19:09:18.144287 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.144267    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqwmz\" (UniqueName: \"kubernetes.io/projected/c7f097a9-49d1-458f-8f26-ee2c929874a8-kube-api-access-vqwmz\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:09:18.144401 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.144292    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5602c815-e107-4f98-b6e3-780f4beed257-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:09:18.144401 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.144305    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zcpg\" (UniqueName: \"kubernetes.io/projected/7e1ff79d-5f8f-4bf0-96c3-3dc489e99572-kube-api-access-2zcpg\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:09:18.144401 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.144318    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7f097a9-49d1-458f-8f26-ee2c929874a8-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:09:18.144401 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.144331    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f097a9-49d1-458f-8f26-ee2c929874a8-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:09:18.144401 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.144343    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfjnf\" (UniqueName: \"kubernetes.io/projected/5602c815-e107-4f98-b6e3-780f4beed257-kube-api-access-jfjnf\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:09:18.144401 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.144356    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ff79d-5f8f-4bf0-96c3-3dc489e99572-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:09:18.149621 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.149600    2580 scope.go:117] "RemoveContainer" containerID="5548105fb6a7c420ae3a9129168fab3a3098bb21b679fae6beffb1618ad95de7"
Jul 24 19:09:18.149880 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:09:18.149861    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5548105fb6a7c420ae3a9129168fab3a3098bb21b679fae6beffb1618ad95de7\": container with ID starting with 5548105fb6a7c420ae3a9129168fab3a3098bb21b679fae6beffb1618ad95de7 not found: ID does not exist" containerID="5548105fb6a7c420ae3a9129168fab3a3098bb21b679fae6beffb1618ad95de7"
Jul 24 19:09:18.149940 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.149887    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5548105fb6a7c420ae3a9129168fab3a3098bb21b679fae6beffb1618ad95de7"} err="failed to get container status \"5548105fb6a7c420ae3a9129168fab3a3098bb21b679fae6beffb1618ad95de7\": rpc error: code = NotFound desc = could not find container \"5548105fb6a7c420ae3a9129168fab3a3098bb21b679fae6beffb1618ad95de7\": container with ID starting with 5548105fb6a7c420ae3a9129168fab3a3098bb21b679fae6beffb1618ad95de7 not found: ID does not exist"
Jul 24 19:09:18.149940 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.149904    2580 scope.go:117] "RemoveContainer" containerID="00f0393bdc1683791f735999f37801c8576cb9e2592c0d2a8d52c1fccb0e7569"
Jul 24 19:09:18.157969 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.157951    2580 scope.go:117] "RemoveContainer" containerID="00f0393bdc1683791f735999f37801c8576cb9e2592c0d2a8d52c1fccb0e7569"
Jul 24 19:09:18.158322 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:09:18.158245    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f0393bdc1683791f735999f37801c8576cb9e2592c0d2a8d52c1fccb0e7569\": container with ID starting with 00f0393bdc1683791f735999f37801c8576cb9e2592c0d2a8d52c1fccb0e7569 not found: ID does not exist" containerID="00f0393bdc1683791f735999f37801c8576cb9e2592c0d2a8d52c1fccb0e7569"
Jul 24 19:09:18.158322 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.158277    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f0393bdc1683791f735999f37801c8576cb9e2592c0d2a8d52c1fccb0e7569"} err="failed to get container status \"00f0393bdc1683791f735999f37801c8576cb9e2592c0d2a8d52c1fccb0e7569\": rpc error: code = NotFound desc = could not find container \"00f0393bdc1683791f735999f37801c8576cb9e2592c0d2a8d52c1fccb0e7569\": container with ID starting with 00f0393bdc1683791f735999f37801c8576cb9e2592c0d2a8d52c1fccb0e7569 not found: ID does not exist"
Jul 24 19:09:18.158594 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.158543    2580 scope.go:117] "RemoveContainer" containerID="830784dba32a5c05b11ec963e0011761352ea092b60f153b2dbd83acd5e5f975"
Jul 24 19:09:18.160644 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.160624    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"]
Jul 24 19:09:18.164408 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.164371    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-889a9e38-7dcccc4889-79hnp"]
Jul 24 19:09:18.167218 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.167201    2580 scope.go:117] "RemoveContainer" containerID="830784dba32a5c05b11ec963e0011761352ea092b60f153b2dbd83acd5e5f975"
Jul 24 19:09:18.167492 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:09:18.167473    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"830784dba32a5c05b11ec963e0011761352ea092b60f153b2dbd83acd5e5f975\": container with ID starting with 830784dba32a5c05b11ec963e0011761352ea092b60f153b2dbd83acd5e5f975 not found: ID does not exist" containerID="830784dba32a5c05b11ec963e0011761352ea092b60f153b2dbd83acd5e5f975"
Jul 24 19:09:18.167555 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.167498    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830784dba32a5c05b11ec963e0011761352ea092b60f153b2dbd83acd5e5f975"} err="failed to get container status \"830784dba32a5c05b11ec963e0011761352ea092b60f153b2dbd83acd5e5f975\": rpc error: code = NotFound desc = could not find container \"830784dba32a5c05b11ec963e0011761352ea092b60f153b2dbd83acd5e5f975\": container with ID starting with 830784dba32a5c05b11ec963e0011761352ea092b60f153b2dbd83acd5e5f975 not found: ID does not exist"
Jul 24 19:09:18.173417 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.173368    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"]
Jul 24 19:09:18.175175 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.175153    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-889a9e38-5cc6cc785b-zn5cr"]
Jul 24 19:09:18.184052 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.184021    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"]
Jul 24 19:09:18.187617 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:18.187598    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-889a9e38-b7fc7c5d5-88kbd"]
Jul 24 19:09:19.196707 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:19.196677    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5602c815-e107-4f98-b6e3-780f4beed257" path="/var/lib/kubelet/pods/5602c815-e107-4f98-b6e3-780f4beed257/volumes"
Jul 24 19:09:19.197078 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:19.197019    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e1ff79d-5f8f-4bf0-96c3-3dc489e99572" path="/var/lib/kubelet/pods/7e1ff79d-5f8f-4bf0-96c3-3dc489e99572/volumes"
Jul 24 19:09:19.197354 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:19.197342    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7f097a9-49d1-458f-8f26-ee2c929874a8" path="/var/lib/kubelet/pods/c7f097a9-49d1-458f-8f26-ee2c929874a8/volumes"
Jul 24 19:09:25.395653 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.395617    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"]
Jul 24 19:09:25.396089 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.396017    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5602c815-e107-4f98-b6e3-780f4beed257" containerName="payload-processing"
Jul 24 19:09:25.396089 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.396039    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5602c815-e107-4f98-b6e3-780f4beed257" containerName="payload-processing"
Jul 24 19:09:25.396089 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.396048    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e1ff79d-5f8f-4bf0-96c3-3dc489e99572" containerName="payload-pre-processing"
Jul 24 19:09:25.396089 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.396054    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e1ff79d-5f8f-4bf0-96c3-3dc489e99572" containerName="payload-pre-processing"
Jul 24 19:09:25.396089 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.396081    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f097a9-49d1-458f-8f26-ee2c929874a8" containerName="maas-api"
Jul 24 19:09:25.396089 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.396087    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f097a9-49d1-458f-8f26-ee2c929874a8" containerName="maas-api"
Jul 24 19:09:25.396286 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.396146    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e1ff79d-5f8f-4bf0-96c3-3dc489e99572" containerName="payload-pre-processing"
Jul 24 19:09:25.396286 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.396154    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f097a9-49d1-458f-8f26-ee2c929874a8" containerName="maas-api"
Jul 24 19:09:25.396286 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.396164    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5602c815-e107-4f98-b6e3-780f4beed257" containerName="payload-processing"
Jul 24 19:09:25.400894 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.400872    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:09:25.403987 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.403960    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ff1b2177-serving-cert\""
Jul 24 19:09:25.412957 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.412931    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"]
Jul 24 19:09:25.510114 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.510079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-openshift-service-ca\") pod \"maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg\" (UID: \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:09:25.510261 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.510155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-maas-api-tls\") pod \"maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg\" (UID: \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:09:25.510261 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.510209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b7ws\" (UniqueName: \"kubernetes.io/projected/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-kube-api-access-9b7ws\") pod \"maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg\" (UID: \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:09:25.611203 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.611160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b7ws\" (UniqueName: \"kubernetes.io/projected/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-kube-api-access-9b7ws\") pod \"maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg\" (UID: \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:09:25.611339 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.611227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-openshift-service-ca\") pod \"maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg\" (UID: \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:09:25.611339 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.611271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-maas-api-tls\") pod \"maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg\" (UID: \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:09:25.611489 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:09:25.611360    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-serving-cert: secret "maas-api-e2e-ait-ff1b2177-serving-cert" not found
Jul 24 19:09:25.611489 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:09:25.611437    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-maas-api-tls podName:3c3077a2-c2ce-436e-b3ae-138f8acba8a5 nodeName:}" failed. No retries permitted until 2026-07-24 19:09:26.11142143 +0000 UTC m=+1765.487080708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-maas-api-tls") pod "maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg" (UID: "3c3077a2-c2ce-436e-b3ae-138f8acba8a5") : secret "maas-api-e2e-ait-ff1b2177-serving-cert" not found
Jul 24 19:09:25.611919 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.611895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-openshift-service-ca\") pod \"maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg\" (UID: \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:09:25.620428 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:25.620405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b7ws\" (UniqueName: \"kubernetes.io/projected/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-kube-api-access-9b7ws\") pod \"maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg\" (UID: \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:09:26.116650 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.116613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-maas-api-tls\") pod \"maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg\" (UID: \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:09:26.118964 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.118941    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-maas-api-tls\") pod \"maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg\" (UID: \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:09:26.230937 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.230901    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"]
Jul 24 19:09:26.234756 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.234738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"
Jul 24 19:09:26.236704 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.236686    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ff1b2177\""
Jul 24 19:09:26.244269 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.244242    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"]
Jul 24 19:09:26.311726 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.311699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:09:26.320274 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.320250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl9j8\" (UniqueName: \"kubernetes.io/projected/3b63c324-1835-4b49-b49c-6d938226e544-kube-api-access-tl9j8\") pod \"payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k\" (UID: \"3b63c324-1835-4b49-b49c-6d938226e544\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"
Jul 24 19:09:26.320363 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.320310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b63c324-1835-4b49-b49c-6d938226e544-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k\" (UID: \"3b63c324-1835-4b49-b49c-6d938226e544\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"
Jul 24 19:09:26.365401 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.365353    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"]
Jul 24 19:09:26.369140 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.369082    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"
Jul 24 19:09:26.371248 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.371225    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ff1b2177-dockercfg-qjrgj\""
Jul 24 19:09:26.380008 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.379972    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"]
Jul 24 19:09:26.420980 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.420949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b63c324-1835-4b49-b49c-6d938226e544-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k\" (UID: \"3b63c324-1835-4b49-b49c-6d938226e544\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"
Jul 24 19:09:26.421423 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.421014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl9j8\" (UniqueName: \"kubernetes.io/projected/3b63c324-1835-4b49-b49c-6d938226e544-kube-api-access-tl9j8\") pod \"payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k\" (UID: \"3b63c324-1835-4b49-b49c-6d938226e544\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"
Jul 24 19:09:26.421423 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.421036    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30ae546a-bd4d-471d-b8fd-febb2679dd05-plugins-config-volume\") pod \"payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs\" (UID: \"30ae546a-bd4d-471d-b8fd-febb2679dd05\") " pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"
Jul 24 19:09:26.421423 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.421069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnnl2\" (UniqueName: \"kubernetes.io/projected/30ae546a-bd4d-471d-b8fd-febb2679dd05-kube-api-access-wnnl2\") pod \"payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs\" (UID: \"30ae546a-bd4d-471d-b8fd-febb2679dd05\") " pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"
Jul 24 19:09:26.421767 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.421746    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b63c324-1835-4b49-b49c-6d938226e544-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k\" (UID: \"3b63c324-1835-4b49-b49c-6d938226e544\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"
Jul 24 19:09:26.432780 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.432756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl9j8\" (UniqueName: \"kubernetes.io/projected/3b63c324-1835-4b49-b49c-6d938226e544-kube-api-access-tl9j8\") pod \"payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k\" (UID: \"3b63c324-1835-4b49-b49c-6d938226e544\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"
Jul 24 19:09:26.455494 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.455470    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"]
Jul 24 19:09:26.457558 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:09:26.457528    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3077a2_c2ce_436e_b3ae_138f8acba8a5.slice/crio-60b6ce4e9e809be5c1feb43ff7e3cf33e1879c58c27f65ca0f9c686f9aeeebc9 WatchSource:0}: Error finding container 60b6ce4e9e809be5c1feb43ff7e3cf33e1879c58c27f65ca0f9c686f9aeeebc9: Status 404 returned error can't find the container with id 60b6ce4e9e809be5c1feb43ff7e3cf33e1879c58c27f65ca0f9c686f9aeeebc9
Jul 24 19:09:26.521839 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.521811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30ae546a-bd4d-471d-b8fd-febb2679dd05-plugins-config-volume\") pod \"payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs\" (UID: \"30ae546a-bd4d-471d-b8fd-febb2679dd05\") " pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"
Jul 24 19:09:26.521960 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.521864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnnl2\" (UniqueName: \"kubernetes.io/projected/30ae546a-bd4d-471d-b8fd-febb2679dd05-kube-api-access-wnnl2\") pod \"payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs\" (UID: \"30ae546a-bd4d-471d-b8fd-febb2679dd05\") " pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"
Jul 24 19:09:26.522503 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.522485    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30ae546a-bd4d-471d-b8fd-febb2679dd05-plugins-config-volume\") pod \"payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs\" (UID: \"30ae546a-bd4d-471d-b8fd-febb2679dd05\") " pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"
Jul 24 19:09:26.529119 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.529096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnnl2\" (UniqueName: \"kubernetes.io/projected/30ae546a-bd4d-471d-b8fd-febb2679dd05-kube-api-access-wnnl2\") pod \"payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs\" (UID: \"30ae546a-bd4d-471d-b8fd-febb2679dd05\") " pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"
Jul 24 19:09:26.546957 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.546937    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"
Jul 24 19:09:26.678013 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.677990    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"]
Jul 24 19:09:26.680266 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:09:26.680238    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b63c324_1835_4b49_b49c_6d938226e544.slice/crio-6c82315dca4b0fbf00ee4bb342fb13d8d2b3102c15791c665c54c84e905fd6fa WatchSource:0}: Error finding container 6c82315dca4b0fbf00ee4bb342fb13d8d2b3102c15791c665c54c84e905fd6fa: Status 404 returned error can't find the container with id 6c82315dca4b0fbf00ee4bb342fb13d8d2b3102c15791c665c54c84e905fd6fa
Jul 24 19:09:26.689677 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.689658    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"
Jul 24 19:09:26.826037 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:26.826014    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"]
Jul 24 19:09:26.831799 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:09:26.831731    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ae546a_bd4d_471d_b8fd_febb2679dd05.slice/crio-fc27b0a894732c738541369cc8eab625b477163921bf581d8a217c1adb298d79 WatchSource:0}: Error finding container fc27b0a894732c738541369cc8eab625b477163921bf581d8a217c1adb298d79: Status 404 returned error can't find the container with id fc27b0a894732c738541369cc8eab625b477163921bf581d8a217c1adb298d79
Jul 24 19:09:27.189928 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:27.189833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg" event={"ID":"3c3077a2-c2ce-436e-b3ae-138f8acba8a5","Type":"ContainerStarted","Data":"2a58c0835a28be450f03ff4b65ea281e4efad6685168076529ec647f3067478f"}
Jul 24 19:09:27.189928 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:27.189889    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg" event={"ID":"3c3077a2-c2ce-436e-b3ae-138f8acba8a5","Type":"ContainerStarted","Data":"60b6ce4e9e809be5c1feb43ff7e3cf33e1879c58c27f65ca0f9c686f9aeeebc9"}
Jul 24 19:09:27.190138 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:27.189934    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:09:27.196568 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:27.196546    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"
Jul 24 19:09:27.196692 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:27.196583    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"
Jul 24 19:09:27.196692 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:27.196599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs" event={"ID":"30ae546a-bd4d-471d-b8fd-febb2679dd05","Type":"ContainerStarted","Data":"5ab5e002c8f64511659a921eb472b863d1906d1ec3dfc66b803acd751e1a07cf"}
Jul 24 19:09:27.196692 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:27.196617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs" event={"ID":"30ae546a-bd4d-471d-b8fd-febb2679dd05","Type":"ContainerStarted","Data":"fc27b0a894732c738541369cc8eab625b477163921bf581d8a217c1adb298d79"}
Jul 24 19:09:27.196692 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:27.196631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k" event={"ID":"3b63c324-1835-4b49-b49c-6d938226e544","Type":"ContainerStarted","Data":"2efc395f8169a5da982ac9b99b4d16a9f4b3d916b4bc8a52b24eaa1a13dc2b4b"}
Jul 24 19:09:27.196692 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:27.196645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k" event={"ID":"3b63c324-1835-4b49-b49c-6d938226e544","Type":"ContainerStarted","Data":"6c82315dca4b0fbf00ee4bb342fb13d8d2b3102c15791c665c54c84e905fd6fa"}
Jul 24 19:09:27.222556 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:27.222514    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg" podStartSLOduration=1.860148826 podStartE2EDuration="2.222500153s" totalImagesPullingTime="362.351327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:09:27.206356498 +0000 UTC m=+1766.582015791" watchObservedRunningTime="2026-07-24 19:09:27.222500153 +0000 UTC m=+1766.598159448"
Jul 24 19:09:27.222669 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:27.222588    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k" podStartSLOduration=1.222582065 podStartE2EDuration="1.222582065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:09:27.221274313 +0000 UTC m=+1766.596933610" watchObservedRunningTime="2026-07-24 19:09:27.222582065 +0000 UTC m=+1766.598241362"
Jul 24 19:09:27.237211 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:27.237165    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs" podStartSLOduration=1.237157561 podStartE2EDuration="1.237157561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:09:27.235151299 +0000 UTC m=+1766.610810598" watchObservedRunningTime="2026-07-24 19:09:27.237157561 +0000 UTC m=+1766.612816891"
Jul 24 19:09:33.204197 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:33.204170    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:09:38.201320 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:38.201283    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"
Jul 24 19:09:38.201878 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:09:38.201354    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"
Jul 24 19:10:01.205485 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.205455    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:10:01.208413 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.208374    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:10:01.490511 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.490436    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"]
Jul 24 19:10:01.490668 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.490631    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg" podUID="3c3077a2-c2ce-436e-b3ae-138f8acba8a5" containerName="maas-api" containerID="cri-o://2a58c0835a28be450f03ff4b65ea281e4efad6685168076529ec647f3067478f" gracePeriod=30
Jul 24 19:10:01.531304 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.531275    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"]
Jul 24 19:10:01.531547 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.531518    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs" podUID="30ae546a-bd4d-471d-b8fd-febb2679dd05" containerName="payload-processing" containerID="cri-o://5ab5e002c8f64511659a921eb472b863d1906d1ec3dfc66b803acd751e1a07cf" gracePeriod=30
Jul 24 19:10:01.539457 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.539144    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"]
Jul 24 19:10:01.539457 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.539361    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k" podUID="3b63c324-1835-4b49-b49c-6d938226e544" containerName="payload-pre-processing" containerID="cri-o://2efc395f8169a5da982ac9b99b4d16a9f4b3d916b4bc8a52b24eaa1a13dc2b4b" gracePeriod=30
Jul 24 19:10:01.795344 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.795315    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"
Jul 24 19:10:01.822214 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.822190    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30ae546a-bd4d-471d-b8fd-febb2679dd05-plugins-config-volume\") pod \"30ae546a-bd4d-471d-b8fd-febb2679dd05\" (UID: \"30ae546a-bd4d-471d-b8fd-febb2679dd05\") "
Jul 24 19:10:01.822373 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.822302    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnnl2\" (UniqueName: \"kubernetes.io/projected/30ae546a-bd4d-471d-b8fd-febb2679dd05-kube-api-access-wnnl2\") pod \"30ae546a-bd4d-471d-b8fd-febb2679dd05\" (UID: \"30ae546a-bd4d-471d-b8fd-febb2679dd05\") "
Jul 24 19:10:01.822648 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.822614    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ae546a-bd4d-471d-b8fd-febb2679dd05-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "30ae546a-bd4d-471d-b8fd-febb2679dd05" (UID: "30ae546a-bd4d-471d-b8fd-febb2679dd05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:10:01.824327 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.824298    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ae546a-bd4d-471d-b8fd-febb2679dd05-kube-api-access-wnnl2" (OuterVolumeSpecName: "kube-api-access-wnnl2") pod "30ae546a-bd4d-471d-b8fd-febb2679dd05" (UID: "30ae546a-bd4d-471d-b8fd-febb2679dd05"). InnerVolumeSpecName "kube-api-access-wnnl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:10:01.837101 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.837084    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"
Jul 24 19:10:01.840085 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.840059    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:10:01.923160 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.923131    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b63c324-1835-4b49-b49c-6d938226e544-plugins-config-volume\") pod \"3b63c324-1835-4b49-b49c-6d938226e544\" (UID: \"3b63c324-1835-4b49-b49c-6d938226e544\") "
Jul 24 19:10:01.923284 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.923226    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-openshift-service-ca\") pod \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\" (UID: \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\") "
Jul 24 19:10:01.923284 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.923255    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-maas-api-tls\") pod \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\" (UID: \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\") "
Jul 24 19:10:01.923368 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.923288    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl9j8\" (UniqueName: \"kubernetes.io/projected/3b63c324-1835-4b49-b49c-6d938226e544-kube-api-access-tl9j8\") pod \"3b63c324-1835-4b49-b49c-6d938226e544\" (UID: \"3b63c324-1835-4b49-b49c-6d938226e544\") "
Jul 24 19:10:01.923368 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.923313    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9b7ws\" (UniqueName: \"kubernetes.io/projected/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-kube-api-access-9b7ws\") pod \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\" (UID: \"3c3077a2-c2ce-436e-b3ae-138f8acba8a5\") "
Jul 24 19:10:01.923567 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.923541    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b63c324-1835-4b49-b49c-6d938226e544-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b63c324-1835-4b49-b49c-6d938226e544" (UID: "3b63c324-1835-4b49-b49c-6d938226e544"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:10:01.923639 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.923594    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30ae546a-bd4d-471d-b8fd-febb2679dd05-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:10:01.923639 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.923613    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnnl2\" (UniqueName: \"kubernetes.io/projected/30ae546a-bd4d-471d-b8fd-febb2679dd05-kube-api-access-wnnl2\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:10:01.923939 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.923914    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3c3077a2-c2ce-436e-b3ae-138f8acba8a5" (UID: "3c3077a2-c2ce-436e-b3ae-138f8acba8a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:10:01.925419 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.925375    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3c3077a2-c2ce-436e-b3ae-138f8acba8a5" (UID: "3c3077a2-c2ce-436e-b3ae-138f8acba8a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:10:01.925419 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.925401    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-kube-api-access-9b7ws" (OuterVolumeSpecName: "kube-api-access-9b7ws") pod "3c3077a2-c2ce-436e-b3ae-138f8acba8a5" (UID: "3c3077a2-c2ce-436e-b3ae-138f8acba8a5"). InnerVolumeSpecName "kube-api-access-9b7ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:10:01.925554 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:01.925444    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b63c324-1835-4b49-b49c-6d938226e544-kube-api-access-tl9j8" (OuterVolumeSpecName: "kube-api-access-tl9j8") pod "3b63c324-1835-4b49-b49c-6d938226e544" (UID: "3b63c324-1835-4b49-b49c-6d938226e544"). InnerVolumeSpecName "kube-api-access-tl9j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:10:02.024007 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.023942    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl9j8\" (UniqueName: \"kubernetes.io/projected/3b63c324-1835-4b49-b49c-6d938226e544-kube-api-access-tl9j8\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:10:02.024007 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.023968    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b7ws\" (UniqueName: \"kubernetes.io/projected/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-kube-api-access-9b7ws\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:10:02.024007 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.023981    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b63c324-1835-4b49-b49c-6d938226e544-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:10:02.024007 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.023994    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:10:02.024007 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.024007    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3077a2-c2ce-436e-b3ae-138f8acba8a5-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:10:02.358942 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.358906    2580 generic.go:356] "Generic (PLEG): container finished" podID="3c3077a2-c2ce-436e-b3ae-138f8acba8a5" containerID="2a58c0835a28be450f03ff4b65ea281e4efad6685168076529ec647f3067478f" exitCode=0
Jul 24 19:10:02.359340 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.358951    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"
Jul 24 19:10:02.359340 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.358981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg" event={"ID":"3c3077a2-c2ce-436e-b3ae-138f8acba8a5","Type":"ContainerDied","Data":"2a58c0835a28be450f03ff4b65ea281e4efad6685168076529ec647f3067478f"}
Jul 24 19:10:02.359340 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.359033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg" event={"ID":"3c3077a2-c2ce-436e-b3ae-138f8acba8a5","Type":"ContainerDied","Data":"60b6ce4e9e809be5c1feb43ff7e3cf33e1879c58c27f65ca0f9c686f9aeeebc9"}
Jul 24 19:10:02.359340 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.359048    2580 scope.go:117] "RemoveContainer" containerID="2a58c0835a28be450f03ff4b65ea281e4efad6685168076529ec647f3067478f"
Jul 24 19:10:02.360454 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.360434    2580 generic.go:356] "Generic (PLEG): container finished" podID="30ae546a-bd4d-471d-b8fd-febb2679dd05" containerID="5ab5e002c8f64511659a921eb472b863d1906d1ec3dfc66b803acd751e1a07cf" exitCode=0
Jul 24 19:10:02.360524 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.360484    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"
Jul 24 19:10:02.360582 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.360520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs" event={"ID":"30ae546a-bd4d-471d-b8fd-febb2679dd05","Type":"ContainerDied","Data":"5ab5e002c8f64511659a921eb472b863d1906d1ec3dfc66b803acd751e1a07cf"}
Jul 24 19:10:02.360635 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.360590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs" event={"ID":"30ae546a-bd4d-471d-b8fd-febb2679dd05","Type":"ContainerDied","Data":"fc27b0a894732c738541369cc8eab625b477163921bf581d8a217c1adb298d79"}
Jul 24 19:10:02.361817 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.361798    2580 generic.go:356] "Generic (PLEG): container finished" podID="3b63c324-1835-4b49-b49c-6d938226e544" containerID="2efc395f8169a5da982ac9b99b4d16a9f4b3d916b4bc8a52b24eaa1a13dc2b4b" exitCode=0
Jul 24 19:10:02.361934 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.361833    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"
Jul 24 19:10:02.361934 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.361854    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k" event={"ID":"3b63c324-1835-4b49-b49c-6d938226e544","Type":"ContainerDied","Data":"2efc395f8169a5da982ac9b99b4d16a9f4b3d916b4bc8a52b24eaa1a13dc2b4b"}
Jul 24 19:10:02.361934 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.361893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k" event={"ID":"3b63c324-1835-4b49-b49c-6d938226e544","Type":"ContainerDied","Data":"6c82315dca4b0fbf00ee4bb342fb13d8d2b3102c15791c665c54c84e905fd6fa"}
Jul 24 19:10:02.369590 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.369564    2580 scope.go:117] "RemoveContainer" containerID="2a58c0835a28be450f03ff4b65ea281e4efad6685168076529ec647f3067478f"
Jul 24 19:10:02.369857 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:10:02.369837    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a58c0835a28be450f03ff4b65ea281e4efad6685168076529ec647f3067478f\": container with ID starting with 2a58c0835a28be450f03ff4b65ea281e4efad6685168076529ec647f3067478f not found: ID does not exist" containerID="2a58c0835a28be450f03ff4b65ea281e4efad6685168076529ec647f3067478f"
Jul 24 19:10:02.369913 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.369864    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a58c0835a28be450f03ff4b65ea281e4efad6685168076529ec647f3067478f"} err="failed to get container status \"2a58c0835a28be450f03ff4b65ea281e4efad6685168076529ec647f3067478f\": rpc error: code = NotFound desc = could not find container \"2a58c0835a28be450f03ff4b65ea281e4efad6685168076529ec647f3067478f\": container with ID starting with 2a58c0835a28be450f03ff4b65ea281e4efad6685168076529ec647f3067478f not found: ID does not exist"
Jul 24 19:10:02.369913 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.369881    2580 scope.go:117] "RemoveContainer" containerID="5ab5e002c8f64511659a921eb472b863d1906d1ec3dfc66b803acd751e1a07cf"
Jul 24 19:10:02.379258 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.379241    2580 scope.go:117] "RemoveContainer" containerID="5ab5e002c8f64511659a921eb472b863d1906d1ec3dfc66b803acd751e1a07cf"
Jul 24 19:10:02.380097 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:10:02.379707    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab5e002c8f64511659a921eb472b863d1906d1ec3dfc66b803acd751e1a07cf\": container with ID starting with 5ab5e002c8f64511659a921eb472b863d1906d1ec3dfc66b803acd751e1a07cf not found: ID does not exist" containerID="5ab5e002c8f64511659a921eb472b863d1906d1ec3dfc66b803acd751e1a07cf"
Jul 24 19:10:02.380097 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.379739    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab5e002c8f64511659a921eb472b863d1906d1ec3dfc66b803acd751e1a07cf"} err="failed to get container status \"5ab5e002c8f64511659a921eb472b863d1906d1ec3dfc66b803acd751e1a07cf\": rpc error: code = NotFound desc = could not find container \"5ab5e002c8f64511659a921eb472b863d1906d1ec3dfc66b803acd751e1a07cf\": container with ID starting with 5ab5e002c8f64511659a921eb472b863d1906d1ec3dfc66b803acd751e1a07cf not found: ID does not exist"
Jul 24 19:10:02.380097 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.379760    2580 scope.go:117] "RemoveContainer" containerID="2efc395f8169a5da982ac9b99b4d16a9f4b3d916b4bc8a52b24eaa1a13dc2b4b"
Jul 24 19:10:02.381955 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.381937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"]
Jul 24 19:10:02.385882 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.385862    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ff1b2177-9fd44b5bc-d9rgg"]
Jul 24 19:10:02.388732 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.388719    2580 scope.go:117] "RemoveContainer" containerID="2efc395f8169a5da982ac9b99b4d16a9f4b3d916b4bc8a52b24eaa1a13dc2b4b"
Jul 24 19:10:02.388977 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:10:02.388963    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2efc395f8169a5da982ac9b99b4d16a9f4b3d916b4bc8a52b24eaa1a13dc2b4b\": container with ID starting with 2efc395f8169a5da982ac9b99b4d16a9f4b3d916b4bc8a52b24eaa1a13dc2b4b not found: ID does not exist" containerID="2efc395f8169a5da982ac9b99b4d16a9f4b3d916b4bc8a52b24eaa1a13dc2b4b"
Jul 24 19:10:02.389015 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.388985    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2efc395f8169a5da982ac9b99b4d16a9f4b3d916b4bc8a52b24eaa1a13dc2b4b"} err="failed to get container status \"2efc395f8169a5da982ac9b99b4d16a9f4b3d916b4bc8a52b24eaa1a13dc2b4b\": rpc error: code = NotFound desc = could not find container \"2efc395f8169a5da982ac9b99b4d16a9f4b3d916b4bc8a52b24eaa1a13dc2b4b\": container with ID starting with 2efc395f8169a5da982ac9b99b4d16a9f4b3d916b4bc8a52b24eaa1a13dc2b4b not found: ID does not exist"
Jul 24 19:10:02.393693 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.393674    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"]
Jul 24 19:10:02.397756 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.397738    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ff1b2177-74c9c9d8cb-jhd2k"]
Jul 24 19:10:02.406764 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.406741    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"]
Jul 24 19:10:02.410405 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:02.410367    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ff1b2177-cbdf99784-5mhrs"]
Jul 24 19:10:03.197208 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:03.197175    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30ae546a-bd4d-471d-b8fd-febb2679dd05" path="/var/lib/kubelet/pods/30ae546a-bd4d-471d-b8fd-febb2679dd05/volumes"
Jul 24 19:10:03.197569 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:03.197555    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b63c324-1835-4b49-b49c-6d938226e544" path="/var/lib/kubelet/pods/3b63c324-1835-4b49-b49c-6d938226e544/volumes"
Jul 24 19:10:03.197906 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:03.197893    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c3077a2-c2ce-436e-b3ae-138f8acba8a5" path="/var/lib/kubelet/pods/3c3077a2-c2ce-436e-b3ae-138f8acba8a5/volumes"
Jul 24 19:10:11.698977 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.698941    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"]
Jul 24 19:10:11.699509 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.699319    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3077a2-c2ce-436e-b3ae-138f8acba8a5" containerName="maas-api"
Jul 24 19:10:11.699509 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.699330    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3077a2-c2ce-436e-b3ae-138f8acba8a5" containerName="maas-api"
Jul 24 19:10:11.699509 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.699349    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ae546a-bd4d-471d-b8fd-febb2679dd05" containerName="payload-processing"
Jul 24 19:10:11.699509 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.699355    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ae546a-bd4d-471d-b8fd-febb2679dd05" containerName="payload-processing"
Jul 24 19:10:11.699509 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.699365    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b63c324-1835-4b49-b49c-6d938226e544" containerName="payload-pre-processing"
Jul 24 19:10:11.699509 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.699371    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b63c324-1835-4b49-b49c-6d938226e544" containerName="payload-pre-processing"
Jul 24 19:10:11.699509 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.699447    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b63c324-1835-4b49-b49c-6d938226e544" containerName="payload-pre-processing"
Jul 24 19:10:11.699509 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.699458    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3077a2-c2ce-436e-b3ae-138f8acba8a5" containerName="maas-api"
Jul 24 19:10:11.699509 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.699465    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ae546a-bd4d-471d-b8fd-febb2679dd05" containerName="payload-processing"
Jul 24 19:10:11.702884 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.702859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:11.704860 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.704842    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a93f4b7c-serving-cert\""
Jul 24 19:10:11.714557 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.714531    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"]
Jul 24 19:10:11.807624 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.807591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede90f13-dfcd-4085-a191-d7012b11ace2-maas-api-tls\") pod \"maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj\" (UID: \"ede90f13-dfcd-4085-a191-d7012b11ace2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:11.807766 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.807639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede90f13-dfcd-4085-a191-d7012b11ace2-openshift-service-ca\") pod \"maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj\" (UID: \"ede90f13-dfcd-4085-a191-d7012b11ace2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:11.807766 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.807709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqmzf\" (UniqueName: \"kubernetes.io/projected/ede90f13-dfcd-4085-a191-d7012b11ace2-kube-api-access-rqmzf\") pod \"maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj\" (UID: \"ede90f13-dfcd-4085-a191-d7012b11ace2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:11.908930 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.908897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede90f13-dfcd-4085-a191-d7012b11ace2-maas-api-tls\") pod \"maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj\" (UID: \"ede90f13-dfcd-4085-a191-d7012b11ace2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:11.909100 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.908939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede90f13-dfcd-4085-a191-d7012b11ace2-openshift-service-ca\") pod \"maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj\" (UID: \"ede90f13-dfcd-4085-a191-d7012b11ace2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:11.909100 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.908968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqmzf\" (UniqueName: \"kubernetes.io/projected/ede90f13-dfcd-4085-a191-d7012b11ace2-kube-api-access-rqmzf\") pod \"maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj\" (UID: \"ede90f13-dfcd-4085-a191-d7012b11ace2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:11.909220 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:10:11.909132    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-serving-cert: secret "maas-api-e2e-derive-a93f4b7c-serving-cert" not found
Jul 24 19:10:11.909220 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:10:11.909191    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede90f13-dfcd-4085-a191-d7012b11ace2-maas-api-tls podName:ede90f13-dfcd-4085-a191-d7012b11ace2 nodeName:}" failed. No retries permitted until 2026-07-24 19:10:12.409173332 +0000 UTC m=+1811.784832624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ede90f13-dfcd-4085-a191-d7012b11ace2-maas-api-tls") pod "maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj" (UID: "ede90f13-dfcd-4085-a191-d7012b11ace2") : secret "maas-api-e2e-derive-a93f4b7c-serving-cert" not found
Jul 24 19:10:11.909719 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.909695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede90f13-dfcd-4085-a191-d7012b11ace2-openshift-service-ca\") pod \"maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj\" (UID: \"ede90f13-dfcd-4085-a191-d7012b11ace2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:11.918974 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:11.918952    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqmzf\" (UniqueName: \"kubernetes.io/projected/ede90f13-dfcd-4085-a191-d7012b11ace2-kube-api-access-rqmzf\") pod \"maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj\" (UID: \"ede90f13-dfcd-4085-a191-d7012b11ace2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:12.299823 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.299787    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"]
Jul 24 19:10:12.303807 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.303783    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"
Jul 24 19:10:12.305916 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.305895    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a93f4b7c\""
Jul 24 19:10:12.306041 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.306010    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a93f4b7c-dockercfg-mvfvk\""
Jul 24 19:10:12.314075 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.314052    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"]
Jul 24 19:10:12.343728 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.343705    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"]
Jul 24 19:10:12.347673 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.347657    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"
Jul 24 19:10:12.356482 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.356462    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"]
Jul 24 19:10:12.414889 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.414850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1ba34e-aa5f-4a62-b951-71db156d7c72-plugins-config-volume\") pod \"payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw\" (UID: \"0c1ba34e-aa5f-4a62-b951-71db156d7c72\") " pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"
Jul 24 19:10:12.415061 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.414900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccjsm\" (UniqueName: \"kubernetes.io/projected/d919474d-6e7c-4e2c-88b1-edf8f4331c14-kube-api-access-ccjsm\") pod \"payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff\" (UID: \"d919474d-6e7c-4e2c-88b1-edf8f4331c14\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"
Jul 24 19:10:12.415061 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.414935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cjrg\" (UniqueName: \"kubernetes.io/projected/0c1ba34e-aa5f-4a62-b951-71db156d7c72-kube-api-access-4cjrg\") pod \"payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw\" (UID: \"0c1ba34e-aa5f-4a62-b951-71db156d7c72\") " pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"
Jul 24 19:10:12.415061 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.414963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d919474d-6e7c-4e2c-88b1-edf8f4331c14-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff\" (UID: \"d919474d-6e7c-4e2c-88b1-edf8f4331c14\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"
Jul 24 19:10:12.415061 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.414999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede90f13-dfcd-4085-a191-d7012b11ace2-maas-api-tls\") pod \"maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj\" (UID: \"ede90f13-dfcd-4085-a191-d7012b11ace2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:12.417332 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.417307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede90f13-dfcd-4085-a191-d7012b11ace2-maas-api-tls\") pod \"maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj\" (UID: \"ede90f13-dfcd-4085-a191-d7012b11ace2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:12.515857 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.515831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1ba34e-aa5f-4a62-b951-71db156d7c72-plugins-config-volume\") pod \"payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw\" (UID: \"0c1ba34e-aa5f-4a62-b951-71db156d7c72\") " pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"
Jul 24 19:10:12.516017 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.515881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccjsm\" (UniqueName: \"kubernetes.io/projected/d919474d-6e7c-4e2c-88b1-edf8f4331c14-kube-api-access-ccjsm\") pod \"payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff\" (UID: \"d919474d-6e7c-4e2c-88b1-edf8f4331c14\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"
Jul 24 19:10:12.516017 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.515909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cjrg\" (UniqueName: \"kubernetes.io/projected/0c1ba34e-aa5f-4a62-b951-71db156d7c72-kube-api-access-4cjrg\") pod \"payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw\" (UID: \"0c1ba34e-aa5f-4a62-b951-71db156d7c72\") " pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"
Jul 24 19:10:12.516017 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.515936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d919474d-6e7c-4e2c-88b1-edf8f4331c14-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff\" (UID: \"d919474d-6e7c-4e2c-88b1-edf8f4331c14\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"
Jul 24 19:10:12.516653 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.516633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1ba34e-aa5f-4a62-b951-71db156d7c72-plugins-config-volume\") pod \"payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw\" (UID: \"0c1ba34e-aa5f-4a62-b951-71db156d7c72\") " pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"
Jul 24 19:10:12.516747 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.516729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d919474d-6e7c-4e2c-88b1-edf8f4331c14-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff\" (UID: \"d919474d-6e7c-4e2c-88b1-edf8f4331c14\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"
Jul 24 19:10:12.528399 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.525106    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccjsm\" (UniqueName: \"kubernetes.io/projected/d919474d-6e7c-4e2c-88b1-edf8f4331c14-kube-api-access-ccjsm\") pod \"payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff\" (UID: \"d919474d-6e7c-4e2c-88b1-edf8f4331c14\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"
Jul 24 19:10:12.529655 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.529628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cjrg\" (UniqueName: \"kubernetes.io/projected/0c1ba34e-aa5f-4a62-b951-71db156d7c72-kube-api-access-4cjrg\") pod \"payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw\" (UID: \"0c1ba34e-aa5f-4a62-b951-71db156d7c72\") " pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"
Jul 24 19:10:12.615592 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.615569    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"
Jul 24 19:10:12.618153 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.618128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:12.658316 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.658284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"
Jul 24 19:10:12.782211 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.782150    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"]
Jul 24 19:10:12.786357 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:10:12.786321    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd919474d_6e7c_4e2c_88b1_edf8f4331c14.slice/crio-2b68f0369236bced2a690db9a1442d17cf4cb0fe1adaaab8a54a792751350253 WatchSource:0}: Error finding container 2b68f0369236bced2a690db9a1442d17cf4cb0fe1adaaab8a54a792751350253: Status 404 returned error can't find the container with id 2b68f0369236bced2a690db9a1442d17cf4cb0fe1adaaab8a54a792751350253
Jul 24 19:10:12.803194 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.803149    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"]
Jul 24 19:10:12.805095 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:10:12.805068    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede90f13_dfcd_4085_a191_d7012b11ace2.slice/crio-8ade7c8f74415693d49a433fe2beb0d0777dc604056724d8a6f6010250753728 WatchSource:0}: Error finding container 8ade7c8f74415693d49a433fe2beb0d0777dc604056724d8a6f6010250753728: Status 404 returned error can't find the container with id 8ade7c8f74415693d49a433fe2beb0d0777dc604056724d8a6f6010250753728
Jul 24 19:10:12.830715 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:12.830622    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"]
Jul 24 19:10:12.832739 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:10:12.832714    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1ba34e_aa5f_4a62_b951_71db156d7c72.slice/crio-9aa92868a87c6752bc12046521484218c281b5cbf547f4896d410c705862257f WatchSource:0}: Error finding container 9aa92868a87c6752bc12046521484218c281b5cbf547f4896d410c705862257f: Status 404 returned error can't find the container with id 9aa92868a87c6752bc12046521484218c281b5cbf547f4896d410c705862257f
Jul 24 19:10:13.415620 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:13.415585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj" event={"ID":"ede90f13-dfcd-4085-a191-d7012b11ace2","Type":"ContainerStarted","Data":"5637beb4ea8278a117e7211e6ab4e6a468e24543ab64aaa2c46318b4081b455a"}
Jul 24 19:10:13.415778 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:13.415759    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:13.415863 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:13.415800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj" event={"ID":"ede90f13-dfcd-4085-a191-d7012b11ace2","Type":"ContainerStarted","Data":"8ade7c8f74415693d49a433fe2beb0d0777dc604056724d8a6f6010250753728"}
Jul 24 19:10:13.417002 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:13.416981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff" event={"ID":"d919474d-6e7c-4e2c-88b1-edf8f4331c14","Type":"ContainerStarted","Data":"88459ff60d0caed18cfd30c516870128863b1ad6a010c82e52d588570e5de2a5"}
Jul 24 19:10:13.417109 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:13.417019    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff" event={"ID":"d919474d-6e7c-4e2c-88b1-edf8f4331c14","Type":"ContainerStarted","Data":"2b68f0369236bced2a690db9a1442d17cf4cb0fe1adaaab8a54a792751350253"}
Jul 24 19:10:13.417109 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:13.417067    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"
Jul 24 19:10:13.418571 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:13.418548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw" event={"ID":"0c1ba34e-aa5f-4a62-b951-71db156d7c72","Type":"ContainerStarted","Data":"2a217f1ed74bbb2bb4642528ea4c354c5a2a465888f1f84a26dd58ef0d53ba51"}
Jul 24 19:10:13.418674 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:13.418586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw" event={"ID":"0c1ba34e-aa5f-4a62-b951-71db156d7c72","Type":"ContainerStarted","Data":"9aa92868a87c6752bc12046521484218c281b5cbf547f4896d410c705862257f"}
Jul 24 19:10:13.418674 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:13.418642    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"
Jul 24 19:10:13.435203 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:13.435140    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj" podStartSLOduration=1.904263525 podStartE2EDuration="2.435122875s" totalImagesPullingTime="530.85935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:10:13.431976721 +0000 UTC m=+1812.807636016" watchObservedRunningTime="2026-07-24 19:10:13.435122875 +0000 UTC m=+1812.810782173"
Jul 24 19:10:13.449273 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:13.449230    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw" podStartSLOduration=1.449216026 podStartE2EDuration="1.449216026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:10:13.446409731 +0000 UTC m=+1812.822069022" watchObservedRunningTime="2026-07-24 19:10:13.449216026 +0000 UTC m=+1812.824875322"
Jul 24 19:10:13.465607 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:13.465558    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff" podStartSLOduration=1.465541124 podStartE2EDuration="1.465541124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:10:13.463569683 +0000 UTC m=+1812.839228982" watchObservedRunningTime="2026-07-24 19:10:13.465541124 +0000 UTC m=+1812.841200421"
Jul 24 19:10:19.429372 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:19.429340    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:24.425559 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:24.425528    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"
Jul 24 19:10:24.425985 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:24.425611    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"
Jul 24 19:10:30.836523 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:30.836488    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"]
Jul 24 19:10:30.837002 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:30.836694    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj" podUID="ede90f13-dfcd-4085-a191-d7012b11ace2" containerName="maas-api" containerID="cri-o://5637beb4ea8278a117e7211e6ab4e6a468e24543ab64aaa2c46318b4081b455a" gracePeriod=30
Jul 24 19:10:30.885477 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:30.885450    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"]
Jul 24 19:10:30.885683 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:30.885641    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw" podUID="0c1ba34e-aa5f-4a62-b951-71db156d7c72" containerName="payload-processing" containerID="cri-o://2a217f1ed74bbb2bb4642528ea4c354c5a2a465888f1f84a26dd58ef0d53ba51" gracePeriod=30
Jul 24 19:10:30.892257 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:30.892229    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"]
Jul 24 19:10:30.892494 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:30.892464    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff" podUID="d919474d-6e7c-4e2c-88b1-edf8f4331c14" containerName="payload-pre-processing" containerID="cri-o://88459ff60d0caed18cfd30c516870128863b1ad6a010c82e52d588570e5de2a5" gracePeriod=30
Jul 24 19:10:31.181108 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.181083    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"
Jul 24 19:10:31.201508 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.201477    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:31.205970 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.205947    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"
Jul 24 19:10:31.282781 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.282747    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede90f13-dfcd-4085-a191-d7012b11ace2-maas-api-tls\") pod \"ede90f13-dfcd-4085-a191-d7012b11ace2\" (UID: \"ede90f13-dfcd-4085-a191-d7012b11ace2\") "
Jul 24 19:10:31.282937 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.282792    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede90f13-dfcd-4085-a191-d7012b11ace2-openshift-service-ca\") pod \"ede90f13-dfcd-4085-a191-d7012b11ace2\" (UID: \"ede90f13-dfcd-4085-a191-d7012b11ace2\") "
Jul 24 19:10:31.282937 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.282825    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccjsm\" (UniqueName: \"kubernetes.io/projected/d919474d-6e7c-4e2c-88b1-edf8f4331c14-kube-api-access-ccjsm\") pod \"d919474d-6e7c-4e2c-88b1-edf8f4331c14\" (UID: \"d919474d-6e7c-4e2c-88b1-edf8f4331c14\") "
Jul 24 19:10:31.282937 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.282845    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d919474d-6e7c-4e2c-88b1-edf8f4331c14-plugins-config-volume\") pod \"d919474d-6e7c-4e2c-88b1-edf8f4331c14\" (UID: \"d919474d-6e7c-4e2c-88b1-edf8f4331c14\") "
Jul 24 19:10:31.282937 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.282883    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cjrg\" (UniqueName: \"kubernetes.io/projected/0c1ba34e-aa5f-4a62-b951-71db156d7c72-kube-api-access-4cjrg\") pod \"0c1ba34e-aa5f-4a62-b951-71db156d7c72\" (UID: \"0c1ba34e-aa5f-4a62-b951-71db156d7c72\") "
Jul 24 19:10:31.283169 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.282962    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqmzf\" (UniqueName: \"kubernetes.io/projected/ede90f13-dfcd-4085-a191-d7012b11ace2-kube-api-access-rqmzf\") pod \"ede90f13-dfcd-4085-a191-d7012b11ace2\" (UID: \"ede90f13-dfcd-4085-a191-d7012b11ace2\") "
Jul 24 19:10:31.283169 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.283006    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1ba34e-aa5f-4a62-b951-71db156d7c72-plugins-config-volume\") pod \"0c1ba34e-aa5f-4a62-b951-71db156d7c72\" (UID: \"0c1ba34e-aa5f-4a62-b951-71db156d7c72\") "
Jul 24 19:10:31.283276 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.283229    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d919474d-6e7c-4e2c-88b1-edf8f4331c14-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d919474d-6e7c-4e2c-88b1-edf8f4331c14" (UID: "d919474d-6e7c-4e2c-88b1-edf8f4331c14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:10:31.284209 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.283628    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1ba34e-aa5f-4a62-b951-71db156d7c72-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c1ba34e-aa5f-4a62-b951-71db156d7c72" (UID: "0c1ba34e-aa5f-4a62-b951-71db156d7c72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:10:31.284209 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.284170    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1ba34e-aa5f-4a62-b951-71db156d7c72-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:10:31.284209 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.284191    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d919474d-6e7c-4e2c-88b1-edf8f4331c14-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:10:31.284672 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.284644    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede90f13-dfcd-4085-a191-d7012b11ace2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ede90f13-dfcd-4085-a191-d7012b11ace2" (UID: "ede90f13-dfcd-4085-a191-d7012b11ace2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:10:31.285398 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.285360    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede90f13-dfcd-4085-a191-d7012b11ace2-kube-api-access-rqmzf" (OuterVolumeSpecName: "kube-api-access-rqmzf") pod "ede90f13-dfcd-4085-a191-d7012b11ace2" (UID: "ede90f13-dfcd-4085-a191-d7012b11ace2"). InnerVolumeSpecName "kube-api-access-rqmzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:10:31.285512 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.285473    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1ba34e-aa5f-4a62-b951-71db156d7c72-kube-api-access-4cjrg" (OuterVolumeSpecName: "kube-api-access-4cjrg") pod "0c1ba34e-aa5f-4a62-b951-71db156d7c72" (UID: "0c1ba34e-aa5f-4a62-b951-71db156d7c72"). InnerVolumeSpecName "kube-api-access-4cjrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:10:31.285740 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.285720    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d919474d-6e7c-4e2c-88b1-edf8f4331c14-kube-api-access-ccjsm" (OuterVolumeSpecName: "kube-api-access-ccjsm") pod "d919474d-6e7c-4e2c-88b1-edf8f4331c14" (UID: "d919474d-6e7c-4e2c-88b1-edf8f4331c14"). InnerVolumeSpecName "kube-api-access-ccjsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:10:31.286423 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.286405    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede90f13-dfcd-4085-a191-d7012b11ace2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ede90f13-dfcd-4085-a191-d7012b11ace2" (UID: "ede90f13-dfcd-4085-a191-d7012b11ace2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:10:31.385676 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.385599    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede90f13-dfcd-4085-a191-d7012b11ace2-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:10:31.385676 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.385624    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede90f13-dfcd-4085-a191-d7012b11ace2-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:10:31.385676 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.385637    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccjsm\" (UniqueName: \"kubernetes.io/projected/d919474d-6e7c-4e2c-88b1-edf8f4331c14-kube-api-access-ccjsm\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:10:31.385676 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.385649    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cjrg\" (UniqueName: \"kubernetes.io/projected/0c1ba34e-aa5f-4a62-b951-71db156d7c72-kube-api-access-4cjrg\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:10:31.385676 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.385664    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqmzf\" (UniqueName: \"kubernetes.io/projected/ede90f13-dfcd-4085-a191-d7012b11ace2-kube-api-access-rqmzf\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:10:31.513488 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.513459    2580 generic.go:356] "Generic (PLEG): container finished" podID="0c1ba34e-aa5f-4a62-b951-71db156d7c72" containerID="2a217f1ed74bbb2bb4642528ea4c354c5a2a465888f1f84a26dd58ef0d53ba51" exitCode=0
Jul 24 19:10:31.513623 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.513503    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"
Jul 24 19:10:31.513623 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.513547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw" event={"ID":"0c1ba34e-aa5f-4a62-b951-71db156d7c72","Type":"ContainerDied","Data":"2a217f1ed74bbb2bb4642528ea4c354c5a2a465888f1f84a26dd58ef0d53ba51"}
Jul 24 19:10:31.513623 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.513593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw" event={"ID":"0c1ba34e-aa5f-4a62-b951-71db156d7c72","Type":"ContainerDied","Data":"9aa92868a87c6752bc12046521484218c281b5cbf547f4896d410c705862257f"}
Jul 24 19:10:31.513623 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.513617    2580 scope.go:117] "RemoveContainer" containerID="2a217f1ed74bbb2bb4642528ea4c354c5a2a465888f1f84a26dd58ef0d53ba51"
Jul 24 19:10:31.515038 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.515012    2580 generic.go:356] "Generic (PLEG): container finished" podID="ede90f13-dfcd-4085-a191-d7012b11ace2" containerID="5637beb4ea8278a117e7211e6ab4e6a468e24543ab64aaa2c46318b4081b455a" exitCode=0
Jul 24 19:10:31.515166 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.515048    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj" event={"ID":"ede90f13-dfcd-4085-a191-d7012b11ace2","Type":"ContainerDied","Data":"5637beb4ea8278a117e7211e6ab4e6a468e24543ab64aaa2c46318b4081b455a"}
Jul 24 19:10:31.515166 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.515070    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"
Jul 24 19:10:31.515166 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.515101    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj" event={"ID":"ede90f13-dfcd-4085-a191-d7012b11ace2","Type":"ContainerDied","Data":"8ade7c8f74415693d49a433fe2beb0d0777dc604056724d8a6f6010250753728"}
Jul 24 19:10:31.517099 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.517073    2580 generic.go:356] "Generic (PLEG): container finished" podID="d919474d-6e7c-4e2c-88b1-edf8f4331c14" containerID="88459ff60d0caed18cfd30c516870128863b1ad6a010c82e52d588570e5de2a5" exitCode=0
Jul 24 19:10:31.517204 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.517112    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff" event={"ID":"d919474d-6e7c-4e2c-88b1-edf8f4331c14","Type":"ContainerDied","Data":"88459ff60d0caed18cfd30c516870128863b1ad6a010c82e52d588570e5de2a5"}
Jul 24 19:10:31.517204 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.517149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff" event={"ID":"d919474d-6e7c-4e2c-88b1-edf8f4331c14","Type":"ContainerDied","Data":"2b68f0369236bced2a690db9a1442d17cf4cb0fe1adaaab8a54a792751350253"}
Jul 24 19:10:31.517321 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.517205    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"
Jul 24 19:10:31.525112 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.525094    2580 scope.go:117] "RemoveContainer" containerID="2a217f1ed74bbb2bb4642528ea4c354c5a2a465888f1f84a26dd58ef0d53ba51"
Jul 24 19:10:31.525565 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:10:31.525539    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a217f1ed74bbb2bb4642528ea4c354c5a2a465888f1f84a26dd58ef0d53ba51\": container with ID starting with 2a217f1ed74bbb2bb4642528ea4c354c5a2a465888f1f84a26dd58ef0d53ba51 not found: ID does not exist" containerID="2a217f1ed74bbb2bb4642528ea4c354c5a2a465888f1f84a26dd58ef0d53ba51"
Jul 24 19:10:31.525671 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.525574    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a217f1ed74bbb2bb4642528ea4c354c5a2a465888f1f84a26dd58ef0d53ba51"} err="failed to get container status \"2a217f1ed74bbb2bb4642528ea4c354c5a2a465888f1f84a26dd58ef0d53ba51\": rpc error: code = NotFound desc = could not find container \"2a217f1ed74bbb2bb4642528ea4c354c5a2a465888f1f84a26dd58ef0d53ba51\": container with ID starting with 2a217f1ed74bbb2bb4642528ea4c354c5a2a465888f1f84a26dd58ef0d53ba51 not found: ID does not exist"
Jul 24 19:10:31.525671 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.525595    2580 scope.go:117] "RemoveContainer" containerID="5637beb4ea8278a117e7211e6ab4e6a468e24543ab64aaa2c46318b4081b455a"
Jul 24 19:10:31.533710 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.533695    2580 scope.go:117] "RemoveContainer" containerID="5637beb4ea8278a117e7211e6ab4e6a468e24543ab64aaa2c46318b4081b455a"
Jul 24 19:10:31.533965 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:10:31.533947    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5637beb4ea8278a117e7211e6ab4e6a468e24543ab64aaa2c46318b4081b455a\": container with ID starting with 5637beb4ea8278a117e7211e6ab4e6a468e24543ab64aaa2c46318b4081b455a not found: ID does not exist" containerID="5637beb4ea8278a117e7211e6ab4e6a468e24543ab64aaa2c46318b4081b455a"
Jul 24 19:10:31.534012 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.533971    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5637beb4ea8278a117e7211e6ab4e6a468e24543ab64aaa2c46318b4081b455a"} err="failed to get container status \"5637beb4ea8278a117e7211e6ab4e6a468e24543ab64aaa2c46318b4081b455a\": rpc error: code = NotFound desc = could not find container \"5637beb4ea8278a117e7211e6ab4e6a468e24543ab64aaa2c46318b4081b455a\": container with ID starting with 5637beb4ea8278a117e7211e6ab4e6a468e24543ab64aaa2c46318b4081b455a not found: ID does not exist"
Jul 24 19:10:31.534012 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.533987    2580 scope.go:117] "RemoveContainer" containerID="88459ff60d0caed18cfd30c516870128863b1ad6a010c82e52d588570e5de2a5"
Jul 24 19:10:31.539809 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.539785    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"]
Jul 24 19:10:31.541524 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.541503    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a93f4b7c-66cd6d5bdc-j2rvj"]
Jul 24 19:10:31.542457 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.542442    2580 scope.go:117] "RemoveContainer" containerID="88459ff60d0caed18cfd30c516870128863b1ad6a010c82e52d588570e5de2a5"
Jul 24 19:10:31.542700 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:10:31.542684    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88459ff60d0caed18cfd30c516870128863b1ad6a010c82e52d588570e5de2a5\": container with ID starting with 88459ff60d0caed18cfd30c516870128863b1ad6a010c82e52d588570e5de2a5 not found: ID does not exist" containerID="88459ff60d0caed18cfd30c516870128863b1ad6a010c82e52d588570e5de2a5"
Jul 24 19:10:31.542766 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.542706    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88459ff60d0caed18cfd30c516870128863b1ad6a010c82e52d588570e5de2a5"} err="failed to get container status \"88459ff60d0caed18cfd30c516870128863b1ad6a010c82e52d588570e5de2a5\": rpc error: code = NotFound desc = could not find container \"88459ff60d0caed18cfd30c516870128863b1ad6a010c82e52d588570e5de2a5\": container with ID starting with 88459ff60d0caed18cfd30c516870128863b1ad6a010c82e52d588570e5de2a5 not found: ID does not exist"
Jul 24 19:10:31.549674 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.549638    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"]
Jul 24 19:10:31.551556 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.551537    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a93f4b7c-85d8d65b86-sbvkw"]
Jul 24 19:10:31.560520 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.560499    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"]
Jul 24 19:10:31.564068 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:31.564047    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a93f4b7c-5c55fd6996-zctff"]
Jul 24 19:10:33.196502 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:33.196460    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c1ba34e-aa5f-4a62-b951-71db156d7c72" path="/var/lib/kubelet/pods/0c1ba34e-aa5f-4a62-b951-71db156d7c72/volumes"
Jul 24 19:10:33.197003 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:33.196979    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d919474d-6e7c-4e2c-88b1-edf8f4331c14" path="/var/lib/kubelet/pods/d919474d-6e7c-4e2c-88b1-edf8f4331c14/volumes"
Jul 24 19:10:33.197423 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:10:33.197404    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ede90f13-dfcd-4085-a191-d7012b11ace2" path="/var/lib/kubelet/pods/ede90f13-dfcd-4085-a191-d7012b11ace2/volumes"
Jul 24 19:12:15.938471 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:15.938435    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"]
Jul 24 19:12:15.938945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:15.938798    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d919474d-6e7c-4e2c-88b1-edf8f4331c14" containerName="payload-pre-processing"
Jul 24 19:12:15.938945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:15.938809    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d919474d-6e7c-4e2c-88b1-edf8f4331c14" containerName="payload-pre-processing"
Jul 24 19:12:15.938945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:15.938827    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c1ba34e-aa5f-4a62-b951-71db156d7c72" containerName="payload-processing"
Jul 24 19:12:15.938945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:15.938833    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c1ba34e-aa5f-4a62-b951-71db156d7c72" containerName="payload-processing"
Jul 24 19:12:15.938945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:15.938844    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede90f13-dfcd-4085-a191-d7012b11ace2" containerName="maas-api"
Jul 24 19:12:15.938945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:15.938849    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede90f13-dfcd-4085-a191-d7012b11ace2" containerName="maas-api"
Jul 24 19:12:15.938945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:15.938915    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d919474d-6e7c-4e2c-88b1-edf8f4331c14" containerName="payload-pre-processing"
Jul 24 19:12:15.938945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:15.938927    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ede90f13-dfcd-4085-a191-d7012b11ace2" containerName="maas-api"
Jul 24 19:12:15.938945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:15.938935    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c1ba34e-aa5f-4a62-b951-71db156d7c72" containerName="payload-processing"
Jul 24 19:12:15.942000 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:15.941981    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:15.947720 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:15.947695    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cf2d8fb2-serving-cert\""
Jul 24 19:12:15.964817 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:15.964795    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"]
Jul 24 19:12:16.062429 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.062369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db7d1409-03a0-4552-a0a6-e24f9648128f-openshift-service-ca\") pod \"maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp\" (UID: \"db7d1409-03a0-4552-a0a6-e24f9648128f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:16.062620 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.062444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm244\" (UniqueName: \"kubernetes.io/projected/db7d1409-03a0-4552-a0a6-e24f9648128f-kube-api-access-xm244\") pod \"maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp\" (UID: \"db7d1409-03a0-4552-a0a6-e24f9648128f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:16.062620 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.062560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db7d1409-03a0-4552-a0a6-e24f9648128f-maas-api-tls\") pod \"maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp\" (UID: \"db7d1409-03a0-4552-a0a6-e24f9648128f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:16.163794 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.163762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm244\" (UniqueName: \"kubernetes.io/projected/db7d1409-03a0-4552-a0a6-e24f9648128f-kube-api-access-xm244\") pod \"maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp\" (UID: \"db7d1409-03a0-4552-a0a6-e24f9648128f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:16.163956 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.163819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db7d1409-03a0-4552-a0a6-e24f9648128f-maas-api-tls\") pod \"maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp\" (UID: \"db7d1409-03a0-4552-a0a6-e24f9648128f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:16.163956 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.163856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db7d1409-03a0-4552-a0a6-e24f9648128f-openshift-service-ca\") pod \"maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp\" (UID: \"db7d1409-03a0-4552-a0a6-e24f9648128f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:16.164083 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:12:16.163971    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-serving-cert: secret "maas-api-e2e-mt-cf2d8fb2-serving-cert" not found
Jul 24 19:12:16.164083 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:12:16.164045    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db7d1409-03a0-4552-a0a6-e24f9648128f-maas-api-tls podName:db7d1409-03a0-4552-a0a6-e24f9648128f nodeName:}" failed. No retries permitted until 2026-07-24 19:12:16.664028202 +0000 UTC m=+1936.039687476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db7d1409-03a0-4552-a0a6-e24f9648128f-maas-api-tls") pod "maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp" (UID: "db7d1409-03a0-4552-a0a6-e24f9648128f") : secret "maas-api-e2e-mt-cf2d8fb2-serving-cert" not found
Jul 24 19:12:16.164479 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.164462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db7d1409-03a0-4552-a0a6-e24f9648128f-openshift-service-ca\") pod \"maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp\" (UID: \"db7d1409-03a0-4552-a0a6-e24f9648128f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:16.176672 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.176637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm244\" (UniqueName: \"kubernetes.io/projected/db7d1409-03a0-4552-a0a6-e24f9648128f-kube-api-access-xm244\") pod \"maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp\" (UID: \"db7d1409-03a0-4552-a0a6-e24f9648128f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:16.668807 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.668770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db7d1409-03a0-4552-a0a6-e24f9648128f-maas-api-tls\") pod \"maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp\" (UID: \"db7d1409-03a0-4552-a0a6-e24f9648128f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:16.671529 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.671504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db7d1409-03a0-4552-a0a6-e24f9648128f-maas-api-tls\") pod \"maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp\" (UID: \"db7d1409-03a0-4552-a0a6-e24f9648128f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:16.680390 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.680354    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"]
Jul 24 19:12:16.683900 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.683884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"
Jul 24 19:12:16.687032 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.687007    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cf2d8fb2\""
Jul 24 19:12:16.687148 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.687034    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cf2d8fb2-dockercfg-wqfw7\""
Jul 24 19:12:16.687804 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.687786    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"]
Jul 24 19:12:16.690913 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.690896    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"
Jul 24 19:12:16.695159 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.695137    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"]
Jul 24 19:12:16.709971 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.709952    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"]
Jul 24 19:12:16.769715 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.769687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtf4q\" (UniqueName: \"kubernetes.io/projected/911c6d92-59ad-4733-a718-862db82bfaba-kube-api-access-mtf4q\") pod \"payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck\" (UID: \"911c6d92-59ad-4733-a718-862db82bfaba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"
Jul 24 19:12:16.769822 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.769723    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knqt4\" (UniqueName: \"kubernetes.io/projected/defaaa74-9b41-44da-a8d1-7924ef4ea139-kube-api-access-knqt4\") pod \"payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr\" (UID: \"defaaa74-9b41-44da-a8d1-7924ef4ea139\") " pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"
Jul 24 19:12:16.769822 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.769757    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defaaa74-9b41-44da-a8d1-7924ef4ea139-plugins-config-volume\") pod \"payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr\" (UID: \"defaaa74-9b41-44da-a8d1-7924ef4ea139\") " pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"
Jul 24 19:12:16.769822 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.769782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911c6d92-59ad-4733-a718-862db82bfaba-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck\" (UID: \"911c6d92-59ad-4733-a718-862db82bfaba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"
Jul 24 19:12:16.852580 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.852553    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:16.870696 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.870669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knqt4\" (UniqueName: \"kubernetes.io/projected/defaaa74-9b41-44da-a8d1-7924ef4ea139-kube-api-access-knqt4\") pod \"payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr\" (UID: \"defaaa74-9b41-44da-a8d1-7924ef4ea139\") " pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"
Jul 24 19:12:16.870809 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.870741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defaaa74-9b41-44da-a8d1-7924ef4ea139-plugins-config-volume\") pod \"payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr\" (UID: \"defaaa74-9b41-44da-a8d1-7924ef4ea139\") " pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"
Jul 24 19:12:16.870809 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.870776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911c6d92-59ad-4733-a718-862db82bfaba-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck\" (UID: \"911c6d92-59ad-4733-a718-862db82bfaba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"
Jul 24 19:12:16.870929 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.870861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtf4q\" (UniqueName: \"kubernetes.io/projected/911c6d92-59ad-4733-a718-862db82bfaba-kube-api-access-mtf4q\") pod \"payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck\" (UID: \"911c6d92-59ad-4733-a718-862db82bfaba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"
Jul 24 19:12:16.871767 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.871744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911c6d92-59ad-4733-a718-862db82bfaba-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck\" (UID: \"911c6d92-59ad-4733-a718-862db82bfaba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"
Jul 24 19:12:16.871876 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.871744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defaaa74-9b41-44da-a8d1-7924ef4ea139-plugins-config-volume\") pod \"payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr\" (UID: \"defaaa74-9b41-44da-a8d1-7924ef4ea139\") " pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"
Jul 24 19:12:16.882082 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.882058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knqt4\" (UniqueName: \"kubernetes.io/projected/defaaa74-9b41-44da-a8d1-7924ef4ea139-kube-api-access-knqt4\") pod \"payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr\" (UID: \"defaaa74-9b41-44da-a8d1-7924ef4ea139\") " pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"
Jul 24 19:12:16.882622 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.882602    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtf4q\" (UniqueName: \"kubernetes.io/projected/911c6d92-59ad-4733-a718-862db82bfaba-kube-api-access-mtf4q\") pod \"payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck\" (UID: \"911c6d92-59ad-4733-a718-862db82bfaba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"
Jul 24 19:12:16.986860 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.986832    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"]
Jul 24 19:12:16.989322 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:12:16.989296    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7d1409_03a0_4552_a0a6_e24f9648128f.slice/crio-6da3d2dcbdcddb0153a3ed20903c9074f78ef5f0f7e12603cdac62728d4c6248 WatchSource:0}: Error finding container 6da3d2dcbdcddb0153a3ed20903c9074f78ef5f0f7e12603cdac62728d4c6248: Status 404 returned error can't find the container with id 6da3d2dcbdcddb0153a3ed20903c9074f78ef5f0f7e12603cdac62728d4c6248
Jul 24 19:12:16.995795 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:16.995775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"
Jul 24 19:12:17.006404 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:17.004130    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"
Jul 24 19:12:17.014461 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:17.014430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp" event={"ID":"db7d1409-03a0-4552-a0a6-e24f9648128f","Type":"ContainerStarted","Data":"6da3d2dcbdcddb0153a3ed20903c9074f78ef5f0f7e12603cdac62728d4c6248"}
Jul 24 19:12:17.147743 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:17.147715    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"]
Jul 24 19:12:17.150727 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:12:17.150698    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911c6d92_59ad_4733_a718_862db82bfaba.slice/crio-d611b4d40ebc48ea5e28e601318155fe8e68429ad834522d6ac97ef288050967 WatchSource:0}: Error finding container d611b4d40ebc48ea5e28e601318155fe8e68429ad834522d6ac97ef288050967: Status 404 returned error can't find the container with id d611b4d40ebc48ea5e28e601318155fe8e68429ad834522d6ac97ef288050967
Jul 24 19:12:17.368179 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:17.368152    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"]
Jul 24 19:12:17.370319 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:12:17.370285    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefaaa74_9b41_44da_a8d1_7924ef4ea139.slice/crio-331589d832689dc36c797f59603ad977390272fb1d53bade4a507b836df6359b WatchSource:0}: Error finding container 331589d832689dc36c797f59603ad977390272fb1d53bade4a507b836df6359b: Status 404 returned error can't find the container with id 331589d832689dc36c797f59603ad977390272fb1d53bade4a507b836df6359b
Jul 24 19:12:18.021231 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:18.021191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr" event={"ID":"defaaa74-9b41-44da-a8d1-7924ef4ea139","Type":"ContainerStarted","Data":"a686727a1bbb9cdca22b1cb964b8e74ceadc92a5acc1ea1884cbc2112283aa36"}
Jul 24 19:12:18.021696 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:18.021260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr" event={"ID":"defaaa74-9b41-44da-a8d1-7924ef4ea139","Type":"ContainerStarted","Data":"331589d832689dc36c797f59603ad977390272fb1d53bade4a507b836df6359b"}
Jul 24 19:12:18.021696 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:18.021295    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"
Jul 24 19:12:18.022795 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:18.022771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp" event={"ID":"db7d1409-03a0-4552-a0a6-e24f9648128f","Type":"ContainerStarted","Data":"ae1368779b60517d866b8358d58bbafb3e28cc18b8154259ba9116fc188f4e5f"}
Jul 24 19:12:18.022902 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:18.022826    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:18.024279 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:18.024251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck" event={"ID":"911c6d92-59ad-4733-a718-862db82bfaba","Type":"ContainerStarted","Data":"df20eaddd214f7080af5492a6a2d558d5fe669bd0b890b4773f948c5e04e9753"}
Jul 24 19:12:18.024391 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:18.024296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck" event={"ID":"911c6d92-59ad-4733-a718-862db82bfaba","Type":"ContainerStarted","Data":"d611b4d40ebc48ea5e28e601318155fe8e68429ad834522d6ac97ef288050967"}
Jul 24 19:12:18.024475 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:18.024373    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"
Jul 24 19:12:18.040147 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:18.040107    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr" podStartSLOduration=2.040095277 podStartE2EDuration="2.040095277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:12:18.038780778 +0000 UTC m=+1937.414440116" watchObservedRunningTime="2026-07-24 19:12:18.040095277 +0000 UTC m=+1937.415754574"
Jul 24 19:12:18.056119 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:18.056079    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp" podStartSLOduration=2.655013494 podStartE2EDuration="3.056067284s" totalImagesPullingTime="401.05379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:12:18.05385403 +0000 UTC m=+1937.429513326" watchObservedRunningTime="2026-07-24 19:12:18.056067284 +0000 UTC m=+1937.431726579"
Jul 24 19:12:18.073034 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:18.072997    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck" podStartSLOduration=2.072985164 podStartE2EDuration="2.072985164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:12:18.070964029 +0000 UTC m=+1937.446623336" watchObservedRunningTime="2026-07-24 19:12:18.072985164 +0000 UTC m=+1937.448644460"
Jul 24 19:12:24.034606 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:24.034576    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:29.031441 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:29.031366    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck" podUID="911c6d92-59ad-4733-a718-862db82bfaba" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.52:9004: connect: connection refused"
Jul 24 19:12:29.032013 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:29.031986    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"
Jul 24 19:12:37.794201 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:37.794165    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"]
Jul 24 19:12:37.794652 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:37.794363    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp" podUID="db7d1409-03a0-4552-a0a6-e24f9648128f" containerName="maas-api" containerID="cri-o://ae1368779b60517d866b8358d58bbafb3e28cc18b8154259ba9116fc188f4e5f" gracePeriod=30
Jul 24 19:12:37.846126 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:37.846097    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"]
Jul 24 19:12:37.846454 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:37.846406    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr" podUID="defaaa74-9b41-44da-a8d1-7924ef4ea139" containerName="payload-processing" containerID="cri-o://a686727a1bbb9cdca22b1cb964b8e74ceadc92a5acc1ea1884cbc2112283aa36" gracePeriod=30
Jul 24 19:12:37.853639 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:37.853346    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"]
Jul 24 19:12:37.853639 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:37.853575    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck" podUID="911c6d92-59ad-4733-a718-862db82bfaba" containerName="payload-pre-processing" containerID="cri-o://df20eaddd214f7080af5492a6a2d558d5fe669bd0b890b4773f948c5e04e9753" gracePeriod=30
Jul 24 19:12:37.854558 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:37.854535    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"
Jul 24 19:12:38.128137 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.128094    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"
Jul 24 19:12:38.128594 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.128568    2580 generic.go:356] "Generic (PLEG): container finished" podID="defaaa74-9b41-44da-a8d1-7924ef4ea139" containerID="a686727a1bbb9cdca22b1cb964b8e74ceadc92a5acc1ea1884cbc2112283aa36" exitCode=0
Jul 24 19:12:38.128740 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.128717    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr" event={"ID":"defaaa74-9b41-44da-a8d1-7924ef4ea139","Type":"ContainerDied","Data":"a686727a1bbb9cdca22b1cb964b8e74ceadc92a5acc1ea1884cbc2112283aa36"}
Jul 24 19:12:38.128817 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.128777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr" event={"ID":"defaaa74-9b41-44da-a8d1-7924ef4ea139","Type":"ContainerDied","Data":"331589d832689dc36c797f59603ad977390272fb1d53bade4a507b836df6359b"}
Jul 24 19:12:38.128817 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.128810    2580 scope.go:117] "RemoveContainer" containerID="a686727a1bbb9cdca22b1cb964b8e74ceadc92a5acc1ea1884cbc2112283aa36"
Jul 24 19:12:38.130108 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.130083    2580 generic.go:356] "Generic (PLEG): container finished" podID="db7d1409-03a0-4552-a0a6-e24f9648128f" containerID="ae1368779b60517d866b8358d58bbafb3e28cc18b8154259ba9116fc188f4e5f" exitCode=0
Jul 24 19:12:38.130199 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.130134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp" event={"ID":"db7d1409-03a0-4552-a0a6-e24f9648128f","Type":"ContainerDied","Data":"ae1368779b60517d866b8358d58bbafb3e28cc18b8154259ba9116fc188f4e5f"}
Jul 24 19:12:38.130199 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.130185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp" event={"ID":"db7d1409-03a0-4552-a0a6-e24f9648128f","Type":"ContainerDied","Data":"6da3d2dcbdcddb0153a3ed20903c9074f78ef5f0f7e12603cdac62728d4c6248"}
Jul 24 19:12:38.130301 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.130210    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6da3d2dcbdcddb0153a3ed20903c9074f78ef5f0f7e12603cdac62728d4c6248"
Jul 24 19:12:38.131751 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.131727    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:38.131751 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.131734    2580 generic.go:356] "Generic (PLEG): container finished" podID="911c6d92-59ad-4733-a718-862db82bfaba" containerID="df20eaddd214f7080af5492a6a2d558d5fe669bd0b890b4773f948c5e04e9753" exitCode=0
Jul 24 19:12:38.131916 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.131765    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck" event={"ID":"911c6d92-59ad-4733-a718-862db82bfaba","Type":"ContainerDied","Data":"df20eaddd214f7080af5492a6a2d558d5fe669bd0b890b4773f948c5e04e9753"}
Jul 24 19:12:38.137662 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.137643    2580 scope.go:117] "RemoveContainer" containerID="a686727a1bbb9cdca22b1cb964b8e74ceadc92a5acc1ea1884cbc2112283aa36"
Jul 24 19:12:38.137921 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:12:38.137903    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a686727a1bbb9cdca22b1cb964b8e74ceadc92a5acc1ea1884cbc2112283aa36\": container with ID starting with a686727a1bbb9cdca22b1cb964b8e74ceadc92a5acc1ea1884cbc2112283aa36 not found: ID does not exist" containerID="a686727a1bbb9cdca22b1cb964b8e74ceadc92a5acc1ea1884cbc2112283aa36"
Jul 24 19:12:38.137983 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.137930    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a686727a1bbb9cdca22b1cb964b8e74ceadc92a5acc1ea1884cbc2112283aa36"} err="failed to get container status \"a686727a1bbb9cdca22b1cb964b8e74ceadc92a5acc1ea1884cbc2112283aa36\": rpc error: code = NotFound desc = could not find container \"a686727a1bbb9cdca22b1cb964b8e74ceadc92a5acc1ea1884cbc2112283aa36\": container with ID starting with a686727a1bbb9cdca22b1cb964b8e74ceadc92a5acc1ea1884cbc2112283aa36 not found: ID does not exist"
Jul 24 19:12:38.149715 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.149681    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"
Jul 24 19:12:38.256542 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.256516    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defaaa74-9b41-44da-a8d1-7924ef4ea139-plugins-config-volume\") pod \"defaaa74-9b41-44da-a8d1-7924ef4ea139\" (UID: \"defaaa74-9b41-44da-a8d1-7924ef4ea139\") "
Jul 24 19:12:38.256704 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.256558    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911c6d92-59ad-4733-a718-862db82bfaba-plugins-config-volume\") pod \"911c6d92-59ad-4733-a718-862db82bfaba\" (UID: \"911c6d92-59ad-4733-a718-862db82bfaba\") "
Jul 24 19:12:38.256704 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.256584    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db7d1409-03a0-4552-a0a6-e24f9648128f-openshift-service-ca\") pod \"db7d1409-03a0-4552-a0a6-e24f9648128f\" (UID: \"db7d1409-03a0-4552-a0a6-e24f9648128f\") "
Jul 24 19:12:38.256704 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.256631    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm244\" (UniqueName: \"kubernetes.io/projected/db7d1409-03a0-4552-a0a6-e24f9648128f-kube-api-access-xm244\") pod \"db7d1409-03a0-4552-a0a6-e24f9648128f\" (UID: \"db7d1409-03a0-4552-a0a6-e24f9648128f\") "
Jul 24 19:12:38.256704 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.256657    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knqt4\" (UniqueName: \"kubernetes.io/projected/defaaa74-9b41-44da-a8d1-7924ef4ea139-kube-api-access-knqt4\") pod \"defaaa74-9b41-44da-a8d1-7924ef4ea139\" (UID: \"defaaa74-9b41-44da-a8d1-7924ef4ea139\") "
Jul 24 19:12:38.256704 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.256683    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtf4q\" (UniqueName: \"kubernetes.io/projected/911c6d92-59ad-4733-a718-862db82bfaba-kube-api-access-mtf4q\") pod \"911c6d92-59ad-4733-a718-862db82bfaba\" (UID: \"911c6d92-59ad-4733-a718-862db82bfaba\") "
Jul 24 19:12:38.257001 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.256735    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db7d1409-03a0-4552-a0a6-e24f9648128f-maas-api-tls\") pod \"db7d1409-03a0-4552-a0a6-e24f9648128f\" (UID: \"db7d1409-03a0-4552-a0a6-e24f9648128f\") "
Jul 24 19:12:38.257001 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.256980    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911c6d92-59ad-4733-a718-862db82bfaba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "911c6d92-59ad-4733-a718-862db82bfaba" (UID: "911c6d92-59ad-4733-a718-862db82bfaba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:12:38.257122 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.256989    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defaaa74-9b41-44da-a8d1-7924ef4ea139-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "defaaa74-9b41-44da-a8d1-7924ef4ea139" (UID: "defaaa74-9b41-44da-a8d1-7924ef4ea139"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:12:38.257122 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.257026    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db7d1409-03a0-4552-a0a6-e24f9648128f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "db7d1409-03a0-4552-a0a6-e24f9648128f" (UID: "db7d1409-03a0-4552-a0a6-e24f9648128f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:12:38.257399 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.257360    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defaaa74-9b41-44da-a8d1-7924ef4ea139-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:12:38.257501 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.257411    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911c6d92-59ad-4733-a718-862db82bfaba-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:12:38.257501 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.257426    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db7d1409-03a0-4552-a0a6-e24f9648128f-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:12:38.258839 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.258809    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7d1409-03a0-4552-a0a6-e24f9648128f-kube-api-access-xm244" (OuterVolumeSpecName: "kube-api-access-xm244") pod "db7d1409-03a0-4552-a0a6-e24f9648128f" (UID: "db7d1409-03a0-4552-a0a6-e24f9648128f"). InnerVolumeSpecName "kube-api-access-xm244". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:12:38.258932 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.258891    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defaaa74-9b41-44da-a8d1-7924ef4ea139-kube-api-access-knqt4" (OuterVolumeSpecName: "kube-api-access-knqt4") pod "defaaa74-9b41-44da-a8d1-7924ef4ea139" (UID: "defaaa74-9b41-44da-a8d1-7924ef4ea139"). InnerVolumeSpecName "kube-api-access-knqt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:12:38.258932 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.258904    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db7d1409-03a0-4552-a0a6-e24f9648128f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "db7d1409-03a0-4552-a0a6-e24f9648128f" (UID: "db7d1409-03a0-4552-a0a6-e24f9648128f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:12:38.258932 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.258919    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911c6d92-59ad-4733-a718-862db82bfaba-kube-api-access-mtf4q" (OuterVolumeSpecName: "kube-api-access-mtf4q") pod "911c6d92-59ad-4733-a718-862db82bfaba" (UID: "911c6d92-59ad-4733-a718-862db82bfaba"). InnerVolumeSpecName "kube-api-access-mtf4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:12:38.358511 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.358479    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtf4q\" (UniqueName: \"kubernetes.io/projected/911c6d92-59ad-4733-a718-862db82bfaba-kube-api-access-mtf4q\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:12:38.358511 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.358508    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db7d1409-03a0-4552-a0a6-e24f9648128f-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:12:38.358714 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.358518    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm244\" (UniqueName: \"kubernetes.io/projected/db7d1409-03a0-4552-a0a6-e24f9648128f-kube-api-access-xm244\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:12:38.358714 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:38.358529    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knqt4\" (UniqueName: \"kubernetes.io/projected/defaaa74-9b41-44da-a8d1-7924ef4ea139-kube-api-access-knqt4\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:12:39.138847 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:39.138816    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"
Jul 24 19:12:39.138847 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:39.138832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck" event={"ID":"911c6d92-59ad-4733-a718-862db82bfaba","Type":"ContainerDied","Data":"d611b4d40ebc48ea5e28e601318155fe8e68429ad834522d6ac97ef288050967"}
Jul 24 19:12:39.139434 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:39.138898    2580 scope.go:117] "RemoveContainer" containerID="df20eaddd214f7080af5492a6a2d558d5fe669bd0b890b4773f948c5e04e9753"
Jul 24 19:12:39.139800 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:39.139781    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"
Jul 24 19:12:39.140285 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:39.139939    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"
Jul 24 19:12:39.165291 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:39.165260    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"]
Jul 24 19:12:39.171200 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:39.171175    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cf2d8fb2-58f48dfcf6-lwvck"]
Jul 24 19:12:39.183269 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:39.183244    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"]
Jul 24 19:12:39.185550 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:39.185528    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf2d8fb2-d796c767-blqwp"]
Jul 24 19:12:39.197766 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:39.197737    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="911c6d92-59ad-4733-a718-862db82bfaba" path="/var/lib/kubelet/pods/911c6d92-59ad-4733-a718-862db82bfaba/volumes"
Jul 24 19:12:39.198266 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:39.198249    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db7d1409-03a0-4552-a0a6-e24f9648128f" path="/var/lib/kubelet/pods/db7d1409-03a0-4552-a0a6-e24f9648128f/volumes"
Jul 24 19:12:39.198596 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:39.198580    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"]
Jul 24 19:12:39.198646 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:39.198612    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cf2d8fb2-548755f65d-rtlxr"]
Jul 24 19:12:41.196255 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:12:41.196222    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="defaaa74-9b41-44da-a8d1-7924ef4ea139" path="/var/lib/kubelet/pods/defaaa74-9b41-44da-a8d1-7924ef4ea139/volumes"
Jul 24 19:13:11.288634 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.288601    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"]
Jul 24 19:13:11.289185 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.289068    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911c6d92-59ad-4733-a718-862db82bfaba" containerName="payload-pre-processing"
Jul 24 19:13:11.289185 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.289087    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="911c6d92-59ad-4733-a718-862db82bfaba" containerName="payload-pre-processing"
Jul 24 19:13:11.289185 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.289113    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="defaaa74-9b41-44da-a8d1-7924ef4ea139" containerName="payload-processing"
Jul 24 19:13:11.289185 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.289122    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="defaaa74-9b41-44da-a8d1-7924ef4ea139" containerName="payload-processing"
Jul 24 19:13:11.289185 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.289137    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db7d1409-03a0-4552-a0a6-e24f9648128f" containerName="maas-api"
Jul 24 19:13:11.289185 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.289146    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="db7d1409-03a0-4552-a0a6-e24f9648128f" containerName="maas-api"
Jul 24 19:13:11.289546 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.289240    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="defaaa74-9b41-44da-a8d1-7924ef4ea139" containerName="payload-processing"
Jul 24 19:13:11.289546 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.289254    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="911c6d92-59ad-4733-a718-862db82bfaba" containerName="payload-pre-processing"
Jul 24 19:13:11.289546 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.289269    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="db7d1409-03a0-4552-a0a6-e24f9648128f" containerName="maas-api"
Jul 24 19:13:11.292489 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.292464    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:13:11.294622 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.294603    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-780ac09a-serving-cert\""
Jul 24 19:13:11.303653 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.303625    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"]
Jul 24 19:13:11.445148 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.445120    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-maas-api-tls\") pod \"maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm\" (UID: \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:13:11.445322 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.445167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-openshift-service-ca\") pod \"maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm\" (UID: \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:13:11.445322 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.445193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v5gz\" (UniqueName: \"kubernetes.io/projected/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-kube-api-access-7v5gz\") pod \"maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm\" (UID: \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:13:11.545942 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.545863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-openshift-service-ca\") pod \"maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm\" (UID: \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:13:11.545942 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.545903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v5gz\" (UniqueName: \"kubernetes.io/projected/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-kube-api-access-7v5gz\") pod \"maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm\" (UID: \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:13:11.546161 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.545982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-maas-api-tls\") pod \"maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm\" (UID: \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:13:11.546873 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.546848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-openshift-service-ca\") pod \"maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm\" (UID: \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:13:11.548319 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.548299    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-maas-api-tls\") pod \"maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm\" (UID: \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:13:11.554009 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.553976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v5gz\" (UniqueName: \"kubernetes.io/projected/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-kube-api-access-7v5gz\") pod \"maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm\" (UID: \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:13:11.605009 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.604975    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:13:11.743060 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.742920    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"]
Jul 24 19:13:11.745396 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:13:11.745343    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a0a58a_9d3f_415f_b6a1_77f68a182fdb.slice/crio-c933363f6d64bd6318eb8bf91ebebd236bf20a83aa0b701a4a18cf68c5db5b27 WatchSource:0}: Error finding container c933363f6d64bd6318eb8bf91ebebd236bf20a83aa0b701a4a18cf68c5db5b27: Status 404 returned error can't find the container with id c933363f6d64bd6318eb8bf91ebebd236bf20a83aa0b701a4a18cf68c5db5b27
Jul 24 19:13:11.746884 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.746864    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 19:13:11.916923 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.916889    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"]
Jul 24 19:13:11.921609 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.921588    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"
Jul 24 19:13:11.923730 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.923705    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-780ac09a\""
Jul 24 19:13:11.923861 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.923843    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-780ac09a-dockercfg-w6d2n\""
Jul 24 19:13:11.928802 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.928782    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"]
Jul 24 19:13:11.952852 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.952823    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"]
Jul 24 19:13:11.956208 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.956188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"
Jul 24 19:13:11.967541 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:11.967522    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"]
Jul 24 19:13:12.050369 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.050338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc99745b-e43b-4448-91cb-6b38864c5065-plugins-config-volume\") pod \"payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw\" (UID: \"fc99745b-e43b-4448-91cb-6b38864c5065\") " pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"
Jul 24 19:13:12.050521 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.050376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2lcc\" (UniqueName: \"kubernetes.io/projected/fc99745b-e43b-4448-91cb-6b38864c5065-kube-api-access-b2lcc\") pod \"payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw\" (UID: \"fc99745b-e43b-4448-91cb-6b38864c5065\") " pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"
Jul 24 19:13:12.050606 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.050507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx\" (UID: \"aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"
Jul 24 19:13:12.050606 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.050559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvmh8\" (UniqueName: \"kubernetes.io/projected/aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4-kube-api-access-dvmh8\") pod \"payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx\" (UID: \"aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"
Jul 24 19:13:12.151883 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.151854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc99745b-e43b-4448-91cb-6b38864c5065-plugins-config-volume\") pod \"payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw\" (UID: \"fc99745b-e43b-4448-91cb-6b38864c5065\") " pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"
Jul 24 19:13:12.152065 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.151892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2lcc\" (UniqueName: \"kubernetes.io/projected/fc99745b-e43b-4448-91cb-6b38864c5065-kube-api-access-b2lcc\") pod \"payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw\" (UID: \"fc99745b-e43b-4448-91cb-6b38864c5065\") " pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"
Jul 24 19:13:12.152065 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.151919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx\" (UID: \"aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"
Jul 24 19:13:12.152065 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.151940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvmh8\" (UniqueName: \"kubernetes.io/projected/aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4-kube-api-access-dvmh8\") pod \"payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx\" (UID: \"aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"
Jul 24 19:13:12.152641 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.152620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc99745b-e43b-4448-91cb-6b38864c5065-plugins-config-volume\") pod \"payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw\" (UID: \"fc99745b-e43b-4448-91cb-6b38864c5065\") " pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"
Jul 24 19:13:12.152641 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.152632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx\" (UID: \"aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"
Jul 24 19:13:12.160270 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.160243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvmh8\" (UniqueName: \"kubernetes.io/projected/aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4-kube-api-access-dvmh8\") pod \"payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx\" (UID: \"aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"
Jul 24 19:13:12.160366 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.160250    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2lcc\" (UniqueName: \"kubernetes.io/projected/fc99745b-e43b-4448-91cb-6b38864c5065-kube-api-access-b2lcc\") pod \"payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw\" (UID: \"fc99745b-e43b-4448-91cb-6b38864c5065\") " pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"
Jul 24 19:13:12.233620 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.233592    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"
Jul 24 19:13:12.265582 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.265558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"
Jul 24 19:13:12.303239 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.303204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm" event={"ID":"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb","Type":"ContainerStarted","Data":"630e4f5e4690e658d49a5d1d0b6f691e4eeef1b3393da985b9fcb1a3c44d2b06"}
Jul 24 19:13:12.303637 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.303266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm" event={"ID":"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb","Type":"ContainerStarted","Data":"c933363f6d64bd6318eb8bf91ebebd236bf20a83aa0b701a4a18cf68c5db5b27"}
Jul 24 19:13:12.303637 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.303421    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:13:12.324125 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.323685    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm" podStartSLOduration=0.846189453 podStartE2EDuration="1.323665496s" totalImagesPullingTime="477.476043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:13:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:13:12.323663585 +0000 UTC m=+1991.699322885" watchObservedRunningTime="2026-07-24 19:13:12.323665496 +0000 UTC m=+1991.699324792"
Jul 24 19:13:12.379973 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.379940    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"]
Jul 24 19:13:12.384211 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:13:12.384161    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa13e1e4_0a30_43c7_9de6_d40f7e0f1aa4.slice/crio-3ec176f15bcbf4c71242eca26c571e7217dc18e462564802cce29c2da9d0b587 WatchSource:0}: Error finding container 3ec176f15bcbf4c71242eca26c571e7217dc18e462564802cce29c2da9d0b587: Status 404 returned error can't find the container with id 3ec176f15bcbf4c71242eca26c571e7217dc18e462564802cce29c2da9d0b587
Jul 24 19:13:12.625277 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:12.625248    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"]
Jul 24 19:13:12.627437 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:13:12.627408    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc99745b_e43b_4448_91cb_6b38864c5065.slice/crio-e6e22e23c31d7d0e1fb04a1e4158c789c0c526b8124f92b5d654885df74eddae WatchSource:0}: Error finding container e6e22e23c31d7d0e1fb04a1e4158c789c0c526b8124f92b5d654885df74eddae: Status 404 returned error can't find the container with id e6e22e23c31d7d0e1fb04a1e4158c789c0c526b8124f92b5d654885df74eddae
Jul 24 19:13:13.309204 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:13.309159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx" event={"ID":"aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4","Type":"ContainerStarted","Data":"e0a7d77baba00d0fd7a54cd3b597a6a4c0d98f727d68f67c8d8b46f51f60916d"}
Jul 24 19:13:13.309635 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:13.309227    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"
Jul 24 19:13:13.309635 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:13.309249    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx" event={"ID":"aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4","Type":"ContainerStarted","Data":"3ec176f15bcbf4c71242eca26c571e7217dc18e462564802cce29c2da9d0b587"}
Jul 24 19:13:13.310667 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:13.310639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw" event={"ID":"fc99745b-e43b-4448-91cb-6b38864c5065","Type":"ContainerStarted","Data":"157209c8fc4eb331c3656227029b9c6d3048da955a6ab2d39ef4c916a0be4ff7"}
Jul 24 19:13:13.310787 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:13.310686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw" event={"ID":"fc99745b-e43b-4448-91cb-6b38864c5065","Type":"ContainerStarted","Data":"e6e22e23c31d7d0e1fb04a1e4158c789c0c526b8124f92b5d654885df74eddae"}
Jul 24 19:13:13.324929 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:13.324892    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx" podStartSLOduration=2.324879212 podStartE2EDuration="2.324879212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:13:13.323224968 +0000 UTC m=+1992.698884291" watchObservedRunningTime="2026-07-24 19:13:13.324879212 +0000 UTC m=+1992.700538507"
Jul 24 19:13:13.339606 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:13.339560    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw" podStartSLOduration=2.339546664 podStartE2EDuration="2.339546664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:13:13.337167276 +0000 UTC m=+1992.712826572" watchObservedRunningTime="2026-07-24 19:13:13.339546664 +0000 UTC m=+1992.715205959"
Jul 24 19:13:14.318239 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:14.318195    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"
Jul 24 19:13:18.316765 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:18.316735    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:13:24.320001 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:24.319969    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"
Jul 24 19:13:25.323568 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:13:25.323538    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"
Jul 24 19:14:15.665253 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:15.665217    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"]
Jul 24 19:14:15.665991 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:15.665480    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm" podUID="c5a0a58a-9d3f-415f-b6a1-77f68a182fdb" containerName="maas-api" containerID="cri-o://630e4f5e4690e658d49a5d1d0b6f691e4eeef1b3393da985b9fcb1a3c44d2b06" gracePeriod=30
Jul 24 19:14:15.720313 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:15.720279    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"]
Jul 24 19:14:15.720574 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:15.720529    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw" podUID="fc99745b-e43b-4448-91cb-6b38864c5065" containerName="payload-processing" containerID="cri-o://157209c8fc4eb331c3656227029b9c6d3048da955a6ab2d39ef4c916a0be4ff7" gracePeriod=30
Jul 24 19:14:15.727004 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:15.726966    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"]
Jul 24 19:14:15.727262 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:15.727180    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx" podUID="aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4" containerName="payload-pre-processing" containerID="cri-o://e0a7d77baba00d0fd7a54cd3b597a6a4c0d98f727d68f67c8d8b46f51f60916d" gracePeriod=30
Jul 24 19:14:15.981733 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:15.981694    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:14:16.016958 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.016931    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"
Jul 24 19:14:16.021220 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.021201    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"
Jul 24 19:14:16.078896 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.078867    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2lcc\" (UniqueName: \"kubernetes.io/projected/fc99745b-e43b-4448-91cb-6b38864c5065-kube-api-access-b2lcc\") pod \"fc99745b-e43b-4448-91cb-6b38864c5065\" (UID: \"fc99745b-e43b-4448-91cb-6b38864c5065\") "
Jul 24 19:14:16.079056 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.078934    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4-plugins-config-volume\") pod \"aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4\" (UID: \"aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4\") "
Jul 24 19:14:16.079056 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.078992    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v5gz\" (UniqueName: \"kubernetes.io/projected/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-kube-api-access-7v5gz\") pod \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\" (UID: \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\") "
Jul 24 19:14:16.079056 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.079023    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvmh8\" (UniqueName: \"kubernetes.io/projected/aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4-kube-api-access-dvmh8\") pod \"aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4\" (UID: \"aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4\") "
Jul 24 19:14:16.079056 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.079054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc99745b-e43b-4448-91cb-6b38864c5065-plugins-config-volume\") pod \"fc99745b-e43b-4448-91cb-6b38864c5065\" (UID: \"fc99745b-e43b-4448-91cb-6b38864c5065\") "
Jul 24 19:14:16.079273 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.079099    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-maas-api-tls\") pod \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\" (UID: \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\") "
Jul 24 19:14:16.079273 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.079125    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-openshift-service-ca\") pod \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\" (UID: \"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb\") "
Jul 24 19:14:16.079501 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.079472    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4" (UID: "aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:14:16.079610 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.079537    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc99745b-e43b-4448-91cb-6b38864c5065-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc99745b-e43b-4448-91cb-6b38864c5065" (UID: "fc99745b-e43b-4448-91cb-6b38864c5065"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:14:16.079799 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.079770    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5a0a58a-9d3f-415f-b6a1-77f68a182fdb" (UID: "c5a0a58a-9d3f-415f-b6a1-77f68a182fdb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:14:16.081164 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.081140    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc99745b-e43b-4448-91cb-6b38864c5065-kube-api-access-b2lcc" (OuterVolumeSpecName: "kube-api-access-b2lcc") pod "fc99745b-e43b-4448-91cb-6b38864c5065" (UID: "fc99745b-e43b-4448-91cb-6b38864c5065"). InnerVolumeSpecName "kube-api-access-b2lcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:14:16.081442 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.081414    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4-kube-api-access-dvmh8" (OuterVolumeSpecName: "kube-api-access-dvmh8") pod "aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4" (UID: "aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4"). InnerVolumeSpecName "kube-api-access-dvmh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:14:16.081546 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.081525    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5a0a58a-9d3f-415f-b6a1-77f68a182fdb" (UID: "c5a0a58a-9d3f-415f-b6a1-77f68a182fdb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:14:16.081590 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.081529    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-kube-api-access-7v5gz" (OuterVolumeSpecName: "kube-api-access-7v5gz") pod "c5a0a58a-9d3f-415f-b6a1-77f68a182fdb" (UID: "c5a0a58a-9d3f-415f-b6a1-77f68a182fdb"). InnerVolumeSpecName "kube-api-access-7v5gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:14:16.180204 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.180173    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc99745b-e43b-4448-91cb-6b38864c5065-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:14:16.180204 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.180200    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:14:16.180409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.180213    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:14:16.180409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.180225    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2lcc\" (UniqueName: \"kubernetes.io/projected/fc99745b-e43b-4448-91cb-6b38864c5065-kube-api-access-b2lcc\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:14:16.180409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.180239    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:14:16.180409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.180250    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v5gz\" (UniqueName: \"kubernetes.io/projected/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb-kube-api-access-7v5gz\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:14:16.180409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.180261    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvmh8\" (UniqueName: \"kubernetes.io/projected/aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4-kube-api-access-dvmh8\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:14:16.618215 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.617930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw" event={"ID":"fc99745b-e43b-4448-91cb-6b38864c5065","Type":"ContainerDied","Data":"157209c8fc4eb331c3656227029b9c6d3048da955a6ab2d39ef4c916a0be4ff7"}
Jul 24 19:14:16.618215 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.618044    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"
Jul 24 19:14:16.618215 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.618052    2580 scope.go:117] "RemoveContainer" containerID="157209c8fc4eb331c3656227029b9c6d3048da955a6ab2d39ef4c916a0be4ff7"
Jul 24 19:14:16.618215 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.617945    2580 generic.go:356] "Generic (PLEG): container finished" podID="fc99745b-e43b-4448-91cb-6b38864c5065" containerID="157209c8fc4eb331c3656227029b9c6d3048da955a6ab2d39ef4c916a0be4ff7" exitCode=0
Jul 24 19:14:16.618622 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.618369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw" event={"ID":"fc99745b-e43b-4448-91cb-6b38864c5065","Type":"ContainerDied","Data":"e6e22e23c31d7d0e1fb04a1e4158c789c0c526b8124f92b5d654885df74eddae"}
Jul 24 19:14:16.626165 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.626139    2580 generic.go:356] "Generic (PLEG): container finished" podID="c5a0a58a-9d3f-415f-b6a1-77f68a182fdb" containerID="630e4f5e4690e658d49a5d1d0b6f691e4eeef1b3393da985b9fcb1a3c44d2b06" exitCode=0
Jul 24 19:14:16.626294 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.626172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm" event={"ID":"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb","Type":"ContainerDied","Data":"630e4f5e4690e658d49a5d1d0b6f691e4eeef1b3393da985b9fcb1a3c44d2b06"}
Jul 24 19:14:16.626294 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.626229    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"
Jul 24 19:14:16.626294 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.626257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm" event={"ID":"c5a0a58a-9d3f-415f-b6a1-77f68a182fdb","Type":"ContainerDied","Data":"c933363f6d64bd6318eb8bf91ebebd236bf20a83aa0b701a4a18cf68c5db5b27"}
Jul 24 19:14:16.627840 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.627814    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4" containerID="e0a7d77baba00d0fd7a54cd3b597a6a4c0d98f727d68f67c8d8b46f51f60916d" exitCode=0
Jul 24 19:14:16.627981 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.627855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx" event={"ID":"aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4","Type":"ContainerDied","Data":"e0a7d77baba00d0fd7a54cd3b597a6a4c0d98f727d68f67c8d8b46f51f60916d"}
Jul 24 19:14:16.627981 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.627885    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"
Jul 24 19:14:16.627981 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.627891    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx" event={"ID":"aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4","Type":"ContainerDied","Data":"3ec176f15bcbf4c71242eca26c571e7217dc18e462564802cce29c2da9d0b587"}
Jul 24 19:14:16.636345 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.636325    2580 scope.go:117] "RemoveContainer" containerID="157209c8fc4eb331c3656227029b9c6d3048da955a6ab2d39ef4c916a0be4ff7"
Jul 24 19:14:16.636655 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:14:16.636628    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157209c8fc4eb331c3656227029b9c6d3048da955a6ab2d39ef4c916a0be4ff7\": container with ID starting with 157209c8fc4eb331c3656227029b9c6d3048da955a6ab2d39ef4c916a0be4ff7 not found: ID does not exist" containerID="157209c8fc4eb331c3656227029b9c6d3048da955a6ab2d39ef4c916a0be4ff7"
Jul 24 19:14:16.636768 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.636662    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157209c8fc4eb331c3656227029b9c6d3048da955a6ab2d39ef4c916a0be4ff7"} err="failed to get container status \"157209c8fc4eb331c3656227029b9c6d3048da955a6ab2d39ef4c916a0be4ff7\": rpc error: code = NotFound desc = could not find container \"157209c8fc4eb331c3656227029b9c6d3048da955a6ab2d39ef4c916a0be4ff7\": container with ID starting with 157209c8fc4eb331c3656227029b9c6d3048da955a6ab2d39ef4c916a0be4ff7 not found: ID does not exist"
Jul 24 19:14:16.636768 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.636684    2580 scope.go:117] "RemoveContainer" containerID="630e4f5e4690e658d49a5d1d0b6f691e4eeef1b3393da985b9fcb1a3c44d2b06"
Jul 24 19:14:16.644794 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.644772    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"]
Jul 24 19:14:16.646991 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.646972    2580 scope.go:117] "RemoveContainer" containerID="630e4f5e4690e658d49a5d1d0b6f691e4eeef1b3393da985b9fcb1a3c44d2b06"
Jul 24 19:14:16.647369 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:14:16.647332    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630e4f5e4690e658d49a5d1d0b6f691e4eeef1b3393da985b9fcb1a3c44d2b06\": container with ID starting with 630e4f5e4690e658d49a5d1d0b6f691e4eeef1b3393da985b9fcb1a3c44d2b06 not found: ID does not exist" containerID="630e4f5e4690e658d49a5d1d0b6f691e4eeef1b3393da985b9fcb1a3c44d2b06"
Jul 24 19:14:16.647488 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.647366    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630e4f5e4690e658d49a5d1d0b6f691e4eeef1b3393da985b9fcb1a3c44d2b06"} err="failed to get container status \"630e4f5e4690e658d49a5d1d0b6f691e4eeef1b3393da985b9fcb1a3c44d2b06\": rpc error: code = NotFound desc = could not find container \"630e4f5e4690e658d49a5d1d0b6f691e4eeef1b3393da985b9fcb1a3c44d2b06\": container with ID starting with 630e4f5e4690e658d49a5d1d0b6f691e4eeef1b3393da985b9fcb1a3c44d2b06 not found: ID does not exist"
Jul 24 19:14:16.647488 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.647404    2580 scope.go:117] "RemoveContainer" containerID="e0a7d77baba00d0fd7a54cd3b597a6a4c0d98f727d68f67c8d8b46f51f60916d"
Jul 24 19:14:16.647949 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.647927    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-780ac09a-59b47d8698-ctnnw"]
Jul 24 19:14:16.656857 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.655855    2580 scope.go:117] "RemoveContainer" containerID="e0a7d77baba00d0fd7a54cd3b597a6a4c0d98f727d68f67c8d8b46f51f60916d"
Jul 24 19:14:16.658945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.658566    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"]
Jul 24 19:14:16.659044 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:14:16.659015    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a7d77baba00d0fd7a54cd3b597a6a4c0d98f727d68f67c8d8b46f51f60916d\": container with ID starting with e0a7d77baba00d0fd7a54cd3b597a6a4c0d98f727d68f67c8d8b46f51f60916d not found: ID does not exist" containerID="e0a7d77baba00d0fd7a54cd3b597a6a4c0d98f727d68f67c8d8b46f51f60916d"
Jul 24 19:14:16.659108 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.659045    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a7d77baba00d0fd7a54cd3b597a6a4c0d98f727d68f67c8d8b46f51f60916d"} err="failed to get container status \"e0a7d77baba00d0fd7a54cd3b597a6a4c0d98f727d68f67c8d8b46f51f60916d\": rpc error: code = NotFound desc = could not find container \"e0a7d77baba00d0fd7a54cd3b597a6a4c0d98f727d68f67c8d8b46f51f60916d\": container with ID starting with e0a7d77baba00d0fd7a54cd3b597a6a4c0d98f727d68f67c8d8b46f51f60916d not found: ID does not exist"
Jul 24 19:14:16.659966 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.659946    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-780ac09a-54ff8cf675-5h8cm"]
Jul 24 19:14:16.668075 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.668055    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"]
Jul 24 19:14:16.671739 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:16.671720    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-780ac09a-755894bf56-qvfgx"]
Jul 24 19:14:16.672420 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:14:16.672391    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc99745b_e43b_4448_91cb_6b38864c5065.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc99745b_e43b_4448_91cb_6b38864c5065.slice/crio-e6e22e23c31d7d0e1fb04a1e4158c789c0c526b8124f92b5d654885df74eddae\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa13e1e4_0a30_43c7_9de6_d40f7e0f1aa4.slice/crio-3ec176f15bcbf4c71242eca26c571e7217dc18e462564802cce29c2da9d0b587\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a0a58a_9d3f_415f_b6a1_77f68a182fdb.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a0a58a_9d3f_415f_b6a1_77f68a182fdb.slice/crio-c933363f6d64bd6318eb8bf91ebebd236bf20a83aa0b701a4a18cf68c5db5b27\": RecentStats: unable to find data in memory cache]"
Jul 24 19:14:17.197129 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:17.197093    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4" path="/var/lib/kubelet/pods/aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4/volumes"
Jul 24 19:14:17.197517 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:17.197503    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a0a58a-9d3f-415f-b6a1-77f68a182fdb" path="/var/lib/kubelet/pods/c5a0a58a-9d3f-415f-b6a1-77f68a182fdb/volumes"
Jul 24 19:14:17.197832 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:14:17.197820    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc99745b-e43b-4448-91cb-6b38864c5065" path="/var/lib/kubelet/pods/fc99745b-e43b-4448-91cb-6b38864c5065/volumes"
Jul 24 19:15:01.239923 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:15:01.239893    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:15:01.241152 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:15:01.241133    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:16:10.185635 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.185550    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"]
Jul 24 19:16:10.186214 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.186162    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc99745b-e43b-4448-91cb-6b38864c5065" containerName="payload-processing"
Jul 24 19:16:10.186214 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.186183    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc99745b-e43b-4448-91cb-6b38864c5065" containerName="payload-processing"
Jul 24 19:16:10.186214 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.186202    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a0a58a-9d3f-415f-b6a1-77f68a182fdb" containerName="maas-api"
Jul 24 19:16:10.186214 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.186212    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a0a58a-9d3f-415f-b6a1-77f68a182fdb" containerName="maas-api"
Jul 24 19:16:10.186447 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.186230    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4" containerName="payload-pre-processing"
Jul 24 19:16:10.186447 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.186241    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4" containerName="payload-pre-processing"
Jul 24 19:16:10.186447 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.186341    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a0a58a-9d3f-415f-b6a1-77f68a182fdb" containerName="maas-api"
Jul 24 19:16:10.186447 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.186354    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa13e1e4-0a30-43c7-9de6-d40f7e0f1aa4" containerName="payload-pre-processing"
Jul 24 19:16:10.186447 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.186368    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc99745b-e43b-4448-91cb-6b38864c5065" containerName="payload-processing"
Jul 24 19:16:10.189935 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.189913    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:16:10.192302 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.192278    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2ee838-serving-cert\""
Jul 24 19:16:10.198289 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.198265    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"]
Jul 24 19:16:10.255599 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.255574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r24p\" (UniqueName: \"kubernetes.io/projected/54537b19-47f5-4561-9450-48e1a7e96f42-kube-api-access-6r24p\") pod \"maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7\" (UID: \"54537b19-47f5-4561-9450-48e1a7e96f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:16:10.255724 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.255645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54537b19-47f5-4561-9450-48e1a7e96f42-maas-api-tls\") pod \"maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7\" (UID: \"54537b19-47f5-4561-9450-48e1a7e96f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:16:10.255724 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.255672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54537b19-47f5-4561-9450-48e1a7e96f42-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7\" (UID: \"54537b19-47f5-4561-9450-48e1a7e96f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:16:10.356914 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.356884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r24p\" (UniqueName: \"kubernetes.io/projected/54537b19-47f5-4561-9450-48e1a7e96f42-kube-api-access-6r24p\") pod \"maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7\" (UID: \"54537b19-47f5-4561-9450-48e1a7e96f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:16:10.357065 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.356965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54537b19-47f5-4561-9450-48e1a7e96f42-maas-api-tls\") pod \"maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7\" (UID: \"54537b19-47f5-4561-9450-48e1a7e96f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:16:10.357125 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:16:10.357078    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-serving-cert: secret "maas-api-e2e-inf-a-2ee838-serving-cert" not found
Jul 24 19:16:10.357125 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.357094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54537b19-47f5-4561-9450-48e1a7e96f42-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7\" (UID: \"54537b19-47f5-4561-9450-48e1a7e96f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:16:10.357215 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:16:10.357137    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54537b19-47f5-4561-9450-48e1a7e96f42-maas-api-tls podName:54537b19-47f5-4561-9450-48e1a7e96f42 nodeName:}" failed. No retries permitted until 2026-07-24 19:16:10.857117143 +0000 UTC m=+2170.232776424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54537b19-47f5-4561-9450-48e1a7e96f42-maas-api-tls") pod "maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7" (UID: "54537b19-47f5-4561-9450-48e1a7e96f42") : secret "maas-api-e2e-inf-a-2ee838-serving-cert" not found
Jul 24 19:16:10.357762 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.357743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54537b19-47f5-4561-9450-48e1a7e96f42-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7\" (UID: \"54537b19-47f5-4561-9450-48e1a7e96f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:16:10.371879 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.371852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r24p\" (UniqueName: \"kubernetes.io/projected/54537b19-47f5-4561-9450-48e1a7e96f42-kube-api-access-6r24p\") pod \"maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7\" (UID: \"54537b19-47f5-4561-9450-48e1a7e96f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:16:10.862174 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.862133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54537b19-47f5-4561-9450-48e1a7e96f42-maas-api-tls\") pod \"maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7\" (UID: \"54537b19-47f5-4561-9450-48e1a7e96f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:16:10.864528 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.864509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54537b19-47f5-4561-9450-48e1a7e96f42-maas-api-tls\") pod \"maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7\" (UID: \"54537b19-47f5-4561-9450-48e1a7e96f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:16:10.873890 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.873865    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"]
Jul 24 19:16:10.877451 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.877432    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"
Jul 24 19:16:10.879999 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.879975    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2ee838\""
Jul 24 19:16:10.889722 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.889700    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"]
Jul 24 19:16:10.913948 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.913923    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"]
Jul 24 19:16:10.917213 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.917196    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"
Jul 24 19:16:10.919364 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.919346    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2ee838-dockercfg-w465t\""
Jul 24 19:16:10.928990 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.928969    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"]
Jul 24 19:16:10.963483 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.963458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e54db6-c033-4ffd-887a-ce0e325febb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6\" (UID: \"00e54db6-c033-4ffd-887a-ce0e325febb5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"
Jul 24 19:16:10.963610 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:10.963556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48j64\" (UniqueName: \"kubernetes.io/projected/00e54db6-c033-4ffd-887a-ce0e325febb5-kube-api-access-48j64\") pod \"payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6\" (UID: \"00e54db6-c033-4ffd-887a-ce0e325febb5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"
Jul 24 19:16:11.064534 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.064497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mndtx\" (UniqueName: \"kubernetes.io/projected/56c47452-8e50-47db-9238-1c82b1c79cb8-kube-api-access-mndtx\") pod \"payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7\" (UID: \"56c47452-8e50-47db-9238-1c82b1c79cb8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"
Jul 24 19:16:11.064709 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.064559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e54db6-c033-4ffd-887a-ce0e325febb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6\" (UID: \"00e54db6-c033-4ffd-887a-ce0e325febb5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"
Jul 24 19:16:11.064709 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.064640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c47452-8e50-47db-9238-1c82b1c79cb8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7\" (UID: \"56c47452-8e50-47db-9238-1c82b1c79cb8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"
Jul 24 19:16:11.064709 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.064705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48j64\" (UniqueName: \"kubernetes.io/projected/00e54db6-c033-4ffd-887a-ce0e325febb5-kube-api-access-48j64\") pod \"payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6\" (UID: \"00e54db6-c033-4ffd-887a-ce0e325febb5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"
Jul 24 19:16:11.065295 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.065272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e54db6-c033-4ffd-887a-ce0e325febb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6\" (UID: \"00e54db6-c033-4ffd-887a-ce0e325febb5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"
Jul 24 19:16:11.078211 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.078189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48j64\" (UniqueName: \"kubernetes.io/projected/00e54db6-c033-4ffd-887a-ce0e325febb5-kube-api-access-48j64\") pod \"payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6\" (UID: \"00e54db6-c033-4ffd-887a-ce0e325febb5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"
Jul 24 19:16:11.100644 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.100623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:16:11.165757 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.165724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mndtx\" (UniqueName: \"kubernetes.io/projected/56c47452-8e50-47db-9238-1c82b1c79cb8-kube-api-access-mndtx\") pod \"payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7\" (UID: \"56c47452-8e50-47db-9238-1c82b1c79cb8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"
Jul 24 19:16:11.165930 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.165830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c47452-8e50-47db-9238-1c82b1c79cb8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7\" (UID: \"56c47452-8e50-47db-9238-1c82b1c79cb8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"
Jul 24 19:16:11.166740 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.166714    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c47452-8e50-47db-9238-1c82b1c79cb8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7\" (UID: \"56c47452-8e50-47db-9238-1c82b1c79cb8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"
Jul 24 19:16:11.174204 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.174184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mndtx\" (UniqueName: \"kubernetes.io/projected/56c47452-8e50-47db-9238-1c82b1c79cb8-kube-api-access-mndtx\") pod \"payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7\" (UID: \"56c47452-8e50-47db-9238-1c82b1c79cb8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"
Jul 24 19:16:11.187996 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.187699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"
Jul 24 19:16:11.227207 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.227175    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"
Jul 24 19:16:11.243745 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.243700    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"]
Jul 24 19:16:11.245726 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:16:11.245692    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54537b19_47f5_4561_9450_48e1a7e96f42.slice/crio-29766fa6e79ae1684d9b0fa667e2b507afdcd8ca85b3331a5fb3e533ab29af74 WatchSource:0}: Error finding container 29766fa6e79ae1684d9b0fa667e2b507afdcd8ca85b3331a5fb3e533ab29af74: Status 404 returned error can't find the container with id 29766fa6e79ae1684d9b0fa667e2b507afdcd8ca85b3331a5fb3e533ab29af74
Jul 24 19:16:11.342651 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.342620    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"]
Jul 24 19:16:11.344323 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:16:11.344296    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e54db6_c033_4ffd_887a_ce0e325febb5.slice/crio-df877c36a5736104fa2f7a0c8cd6ace9f1855ee0fffa389237ec07a679a6cbf7 WatchSource:0}: Error finding container df877c36a5736104fa2f7a0c8cd6ace9f1855ee0fffa389237ec07a679a6cbf7: Status 404 returned error can't find the container with id df877c36a5736104fa2f7a0c8cd6ace9f1855ee0fffa389237ec07a679a6cbf7
Jul 24 19:16:11.378327 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:11.378305    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"]
Jul 24 19:16:11.382544 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:16:11.382520    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c47452_8e50_47db_9238_1c82b1c79cb8.slice/crio-13fc6aeb5ea097931671cb0294ac6bf37bc03fc6427ed924b68fa363ef877dbe WatchSource:0}: Error finding container 13fc6aeb5ea097931671cb0294ac6bf37bc03fc6427ed924b68fa363ef877dbe: Status 404 returned error can't find the container with id 13fc6aeb5ea097931671cb0294ac6bf37bc03fc6427ed924b68fa363ef877dbe
Jul 24 19:16:12.177458 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:12.177420    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6" event={"ID":"00e54db6-c033-4ffd-887a-ce0e325febb5","Type":"ContainerStarted","Data":"0e3ee32104a44a6e72de4f5af2d2bcc2d6106268ddfca2fceecbeb69e3f0f362"}
Jul 24 19:16:12.177637 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:12.177485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6" event={"ID":"00e54db6-c033-4ffd-887a-ce0e325febb5","Type":"ContainerStarted","Data":"df877c36a5736104fa2f7a0c8cd6ace9f1855ee0fffa389237ec07a679a6cbf7"}
Jul 24 19:16:12.177637 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:12.177608    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"
Jul 24 19:16:12.178987 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:12.178964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7" event={"ID":"56c47452-8e50-47db-9238-1c82b1c79cb8","Type":"ContainerStarted","Data":"f78aa909d47d2faf23920cb37b5b36770419e45ad45b3cb57fe6d5a197c0e6da"}
Jul 24 19:16:12.179078 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:12.179005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"
Jul 24 19:16:12.179078 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:12.179026    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7" event={"ID":"56c47452-8e50-47db-9238-1c82b1c79cb8","Type":"ContainerStarted","Data":"13fc6aeb5ea097931671cb0294ac6bf37bc03fc6427ed924b68fa363ef877dbe"}
Jul 24 19:16:12.180663 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:12.180635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7" event={"ID":"54537b19-47f5-4561-9450-48e1a7e96f42","Type":"ContainerStarted","Data":"881ec8b8b075cdb55e69dcdf69d93bc1c1586635114eb438614638c6fb71d73a"}
Jul 24 19:16:12.180763 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:12.180683    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:16:12.180763 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:12.180701    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7" event={"ID":"54537b19-47f5-4561-9450-48e1a7e96f42","Type":"ContainerStarted","Data":"29766fa6e79ae1684d9b0fa667e2b507afdcd8ca85b3331a5fb3e533ab29af74"}
Jul 24 19:16:12.198658 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:12.198617    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6" podStartSLOduration=2.198605238 podStartE2EDuration="2.198605238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:16:12.195439694 +0000 UTC m=+2171.571098990" watchObservedRunningTime="2026-07-24 19:16:12.198605238 +0000 UTC m=+2171.574264535"
Jul 24 19:16:12.215539 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:12.215497    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7" podStartSLOduration=2.215486967 podStartE2EDuration="2.215486967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:16:12.214862697 +0000 UTC m=+2171.590521995" watchObservedRunningTime="2026-07-24 19:16:12.215486967 +0000 UTC m=+2171.591146242"
Jul 24 19:16:18.116315 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.116263    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7" podStartSLOduration=7.583848666 podStartE2EDuration="8.116246013s" totalImagesPullingTime="532.397347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:16:12.232698923 +0000 UTC m=+2171.608358220" watchObservedRunningTime="2026-07-24 19:16:18.116246013 +0000 UTC m=+2177.491905309"
Jul 24 19:16:18.116827 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.116528    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"]
Jul 24 19:16:18.120616 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.120591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:16:18.123321 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.123298    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-31c4fa-serving-cert\""
Jul 24 19:16:18.132438 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.132415    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"]
Jul 24 19:16:18.190545 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.190520    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:16:18.223360 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.223330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/558c30f7-d5cd-479c-b3a3-855cc388f392-openshift-service-ca\") pod \"maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk\" (UID: \"558c30f7-d5cd-479c-b3a3-855cc388f392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:16:18.223501 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.223364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd5jp\" (UniqueName: \"kubernetes.io/projected/558c30f7-d5cd-479c-b3a3-855cc388f392-kube-api-access-vd5jp\") pod \"maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk\" (UID: \"558c30f7-d5cd-479c-b3a3-855cc388f392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:16:18.223501 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.223412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/558c30f7-d5cd-479c-b3a3-855cc388f392-maas-api-tls\") pod \"maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk\" (UID: \"558c30f7-d5cd-479c-b3a3-855cc388f392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:16:18.324346 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.324314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/558c30f7-d5cd-479c-b3a3-855cc388f392-maas-api-tls\") pod \"maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk\" (UID: \"558c30f7-d5cd-479c-b3a3-855cc388f392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:16:18.324548 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.324443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/558c30f7-d5cd-479c-b3a3-855cc388f392-openshift-service-ca\") pod \"maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk\" (UID: \"558c30f7-d5cd-479c-b3a3-855cc388f392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:16:18.324548 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.324467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd5jp\" (UniqueName: \"kubernetes.io/projected/558c30f7-d5cd-479c-b3a3-855cc388f392-kube-api-access-vd5jp\") pod \"maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk\" (UID: \"558c30f7-d5cd-479c-b3a3-855cc388f392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:16:18.324548 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:16:18.324479    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-serving-cert: secret "maas-api-e2e-inf-b-31c4fa-serving-cert" not found
Jul 24 19:16:18.324742 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:16:18.324551    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/558c30f7-d5cd-479c-b3a3-855cc388f392-maas-api-tls podName:558c30f7-d5cd-479c-b3a3-855cc388f392 nodeName:}" failed. No retries permitted until 2026-07-24 19:16:18.824529687 +0000 UTC m=+2178.200188968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/558c30f7-d5cd-479c-b3a3-855cc388f392-maas-api-tls") pod "maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk" (UID: "558c30f7-d5cd-479c-b3a3-855cc388f392") : secret "maas-api-e2e-inf-b-31c4fa-serving-cert" not found
Jul 24 19:16:18.325088 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.325068    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/558c30f7-d5cd-479c-b3a3-855cc388f392-openshift-service-ca\") pod \"maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk\" (UID: \"558c30f7-d5cd-479c-b3a3-855cc388f392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:16:18.335079 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.335053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd5jp\" (UniqueName: \"kubernetes.io/projected/558c30f7-d5cd-479c-b3a3-855cc388f392-kube-api-access-vd5jp\") pod \"maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk\" (UID: \"558c30f7-d5cd-479c-b3a3-855cc388f392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:16:18.829269 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.829233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/558c30f7-d5cd-479c-b3a3-855cc388f392-maas-api-tls\") pod \"maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk\" (UID: \"558c30f7-d5cd-479c-b3a3-855cc388f392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:16:18.832009 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.831983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/558c30f7-d5cd-479c-b3a3-855cc388f392-maas-api-tls\") pod \"maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk\" (UID: \"558c30f7-d5cd-479c-b3a3-855cc388f392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:16:18.832261 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.832237    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"]
Jul 24 19:16:18.836152 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.836135    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"
Jul 24 19:16:18.838304 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.838285    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-31c4fa-dockercfg-d75b2\""
Jul 24 19:16:18.838415 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.838294    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-31c4fa\""
Jul 24 19:16:18.849824 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.849801    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"]
Jul 24 19:16:18.854327 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.854303    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"]
Jul 24 19:16:18.857931 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.857911    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"
Jul 24 19:16:18.871925 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.871903    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"]
Jul 24 19:16:18.929732 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.929699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkgq5\" (UniqueName: \"kubernetes.io/projected/e472270b-16f0-4c5a-a1cb-954e367987d0-kube-api-access-dkgq5\") pod \"payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh\" (UID: \"e472270b-16f0-4c5a-a1cb-954e367987d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"
Jul 24 19:16:18.929863 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:18.929741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e472270b-16f0-4c5a-a1cb-954e367987d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh\" (UID: \"e472270b-16f0-4c5a-a1cb-954e367987d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"
Jul 24 19:16:19.031146 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.031103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkgq5\" (UniqueName: \"kubernetes.io/projected/e472270b-16f0-4c5a-a1cb-954e367987d0-kube-api-access-dkgq5\") pod \"payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh\" (UID: \"e472270b-16f0-4c5a-a1cb-954e367987d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"
Jul 24 19:16:19.031306 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.031163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e472270b-16f0-4c5a-a1cb-954e367987d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh\" (UID: \"e472270b-16f0-4c5a-a1cb-954e367987d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"
Jul 24 19:16:19.031306 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.031192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59430d9c-234d-47f9-abbd-82f9c4ec1ba0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5\" (UID: \"59430d9c-234d-47f9-abbd-82f9c4ec1ba0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"
Jul 24 19:16:19.031306 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.031239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q2kj\" (UniqueName: \"kubernetes.io/projected/59430d9c-234d-47f9-abbd-82f9c4ec1ba0-kube-api-access-5q2kj\") pod \"payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5\" (UID: \"59430d9c-234d-47f9-abbd-82f9c4ec1ba0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"
Jul 24 19:16:19.031878 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.031857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e472270b-16f0-4c5a-a1cb-954e367987d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh\" (UID: \"e472270b-16f0-4c5a-a1cb-954e367987d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"
Jul 24 19:16:19.031945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.031927    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:16:19.038443 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.038416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkgq5\" (UniqueName: \"kubernetes.io/projected/e472270b-16f0-4c5a-a1cb-954e367987d0-kube-api-access-dkgq5\") pod \"payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh\" (UID: \"e472270b-16f0-4c5a-a1cb-954e367987d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"
Jul 24 19:16:19.132106 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.132047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59430d9c-234d-47f9-abbd-82f9c4ec1ba0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5\" (UID: \"59430d9c-234d-47f9-abbd-82f9c4ec1ba0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"
Jul 24 19:16:19.132662 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.132125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q2kj\" (UniqueName: \"kubernetes.io/projected/59430d9c-234d-47f9-abbd-82f9c4ec1ba0-kube-api-access-5q2kj\") pod \"payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5\" (UID: \"59430d9c-234d-47f9-abbd-82f9c4ec1ba0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"
Jul 24 19:16:19.132953 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.132930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59430d9c-234d-47f9-abbd-82f9c4ec1ba0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5\" (UID: \"59430d9c-234d-47f9-abbd-82f9c4ec1ba0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"
Jul 24 19:16:19.144578 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.144552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q2kj\" (UniqueName: \"kubernetes.io/projected/59430d9c-234d-47f9-abbd-82f9c4ec1ba0-kube-api-access-5q2kj\") pod \"payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5\" (UID: \"59430d9c-234d-47f9-abbd-82f9c4ec1ba0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"
Jul 24 19:16:19.151200 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.151176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"
Jul 24 19:16:19.168161 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.168135    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"
Jul 24 19:16:19.168764 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.168739    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"]
Jul 24 19:16:19.173244 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:16:19.173220    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558c30f7_d5cd_479c_b3a3_855cc388f392.slice/crio-0a9d7c58f92ea2df1f44e58297a76bf8c8431142703c2ad4c333a833073ebe7e WatchSource:0}: Error finding container 0a9d7c58f92ea2df1f44e58297a76bf8c8431142703c2ad4c333a833073ebe7e: Status 404 returned error can't find the container with id 0a9d7c58f92ea2df1f44e58297a76bf8c8431142703c2ad4c333a833073ebe7e
Jul 24 19:16:19.215426 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.215372    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk" event={"ID":"558c30f7-d5cd-479c-b3a3-855cc388f392","Type":"ContainerStarted","Data":"0a9d7c58f92ea2df1f44e58297a76bf8c8431142703c2ad4c333a833073ebe7e"}
Jul 24 19:16:19.305715 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.305682    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"]
Jul 24 19:16:19.307102 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:16:19.307073    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode472270b_16f0_4c5a_a1cb_954e367987d0.slice/crio-db08eb57529dc1945eb12d69a40d55bcc463732a1139f5ba93ce6d4c3c8ed97a WatchSource:0}: Error finding container db08eb57529dc1945eb12d69a40d55bcc463732a1139f5ba93ce6d4c3c8ed97a: Status 404 returned error can't find the container with id db08eb57529dc1945eb12d69a40d55bcc463732a1139f5ba93ce6d4c3c8ed97a
Jul 24 19:16:19.372951 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:19.372927    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"]
Jul 24 19:16:19.377574 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:16:19.377545    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59430d9c_234d_47f9_abbd_82f9c4ec1ba0.slice/crio-45b36f2c05bfe71924d636752d13b223fedae68bd29171bf036412e4bf937eeb WatchSource:0}: Error finding container 45b36f2c05bfe71924d636752d13b223fedae68bd29171bf036412e4bf937eeb: Status 404 returned error can't find the container with id 45b36f2c05bfe71924d636752d13b223fedae68bd29171bf036412e4bf937eeb
Jul 24 19:16:20.222937 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:20.222893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5" event={"ID":"59430d9c-234d-47f9-abbd-82f9c4ec1ba0","Type":"ContainerStarted","Data":"4e49a9f391a89fd4e84f28cd6cb55916249ad336da94fe8bca5ecf68bd5f5fce"}
Jul 24 19:16:20.223404 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:20.222962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5" event={"ID":"59430d9c-234d-47f9-abbd-82f9c4ec1ba0","Type":"ContainerStarted","Data":"45b36f2c05bfe71924d636752d13b223fedae68bd29171bf036412e4bf937eeb"}
Jul 24 19:16:20.223404 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:20.223118    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"
Jul 24 19:16:20.224437 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:20.224407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh" event={"ID":"e472270b-16f0-4c5a-a1cb-954e367987d0","Type":"ContainerStarted","Data":"75a59ee8883b70ac5807f379d14e0079d0905019b2d1edf65f1571c9f95cb352"}
Jul 24 19:16:20.224557 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:20.224455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh" event={"ID":"e472270b-16f0-4c5a-a1cb-954e367987d0","Type":"ContainerStarted","Data":"db08eb57529dc1945eb12d69a40d55bcc463732a1139f5ba93ce6d4c3c8ed97a"}
Jul 24 19:16:20.224557 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:20.224490    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"
Jul 24 19:16:20.226017 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:20.225988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk" event={"ID":"558c30f7-d5cd-479c-b3a3-855cc388f392","Type":"ContainerStarted","Data":"71e6e18301392bc3610db768e7ff385d7bcca1ac082cf7246f688c9bf7a2ca04"}
Jul 24 19:16:20.226116 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:20.226032    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:16:20.241247 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:20.241202    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5" podStartSLOduration=2.241185678 podStartE2EDuration="2.241185678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:16:20.238563875 +0000 UTC m=+2179.614223175" watchObservedRunningTime="2026-07-24 19:16:20.241185678 +0000 UTC m=+2179.616844972"
Jul 24 19:16:20.254016 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:20.253971    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh" podStartSLOduration=2.253956344 podStartE2EDuration="2.253956344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:16:20.252061527 +0000 UTC m=+2179.627720823" watchObservedRunningTime="2026-07-24 19:16:20.253956344 +0000 UTC m=+2179.629615644"
Jul 24 19:16:20.271618 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:20.271579    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk" podStartSLOduration=1.640272736 podStartE2EDuration="2.271569864s" totalImagesPullingTime="631.297128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:16:20.269770643 +0000 UTC m=+2179.645429936" watchObservedRunningTime="2026-07-24 19:16:20.271569864 +0000 UTC m=+2179.647229160"
Jul 24 19:16:23.187179 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:23.187145    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"
Jul 24 19:16:23.187611 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:23.187219    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"
Jul 24 19:16:26.236820 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:26.236789    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:16:31.234150 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:31.234116    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"
Jul 24 19:16:31.234665 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:16:31.234191    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"
Jul 24 19:17:58.513894 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.513859    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"]
Jul 24 19:17:58.514515 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.514090    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7" podUID="54537b19-47f5-4561-9450-48e1a7e96f42" containerName="maas-api" containerID="cri-o://881ec8b8b075cdb55e69dcdf69d93bc1c1586635114eb438614638c6fb71d73a" gracePeriod=30
Jul 24 19:17:58.574260 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.574226    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"]
Jul 24 19:17:58.574543 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.574492    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7" podUID="56c47452-8e50-47db-9238-1c82b1c79cb8" containerName="payload-processing" containerID="cri-o://f78aa909d47d2faf23920cb37b5b36770419e45ad45b3cb57fe6d5a197c0e6da" gracePeriod=30
Jul 24 19:17:58.584884 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.584856    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"]
Jul 24 19:17:58.585092 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.585070    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6" podUID="00e54db6-c033-4ffd-887a-ce0e325febb5" containerName="payload-pre-processing" containerID="cri-o://0e3ee32104a44a6e72de4f5af2d2bcc2d6106268ddfca2fceecbeb69e3f0f362" gracePeriod=30
Jul 24 19:17:58.716580 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.716448    2580 generic.go:356] "Generic (PLEG): container finished" podID="56c47452-8e50-47db-9238-1c82b1c79cb8" containerID="f78aa909d47d2faf23920cb37b5b36770419e45ad45b3cb57fe6d5a197c0e6da" exitCode=0
Jul 24 19:17:58.716580 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.716554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7" event={"ID":"56c47452-8e50-47db-9238-1c82b1c79cb8","Type":"ContainerDied","Data":"f78aa909d47d2faf23920cb37b5b36770419e45ad45b3cb57fe6d5a197c0e6da"}
Jul 24 19:17:58.718313 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.718289    2580 generic.go:356] "Generic (PLEG): container finished" podID="54537b19-47f5-4561-9450-48e1a7e96f42" containerID="881ec8b8b075cdb55e69dcdf69d93bc1c1586635114eb438614638c6fb71d73a" exitCode=0
Jul 24 19:17:58.718441 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.718349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7" event={"ID":"54537b19-47f5-4561-9450-48e1a7e96f42","Type":"ContainerDied","Data":"881ec8b8b075cdb55e69dcdf69d93bc1c1586635114eb438614638c6fb71d73a"}
Jul 24 19:17:58.719983 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.719957    2580 generic.go:356] "Generic (PLEG): container finished" podID="00e54db6-c033-4ffd-887a-ce0e325febb5" containerID="0e3ee32104a44a6e72de4f5af2d2bcc2d6106268ddfca2fceecbeb69e3f0f362" exitCode=0
Jul 24 19:17:58.720087 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.720017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6" event={"ID":"00e54db6-c033-4ffd-887a-ce0e325febb5","Type":"ContainerDied","Data":"0e3ee32104a44a6e72de4f5af2d2bcc2d6106268ddfca2fceecbeb69e3f0f362"}
Jul 24 19:17:58.842249 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.842215    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:17:58.869755 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.869734    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"
Jul 24 19:17:58.881551 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.881533    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"
Jul 24 19:17:58.987553 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.987524    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54537b19-47f5-4561-9450-48e1a7e96f42-openshift-service-ca\") pod \"54537b19-47f5-4561-9450-48e1a7e96f42\" (UID: \"54537b19-47f5-4561-9450-48e1a7e96f42\") "
Jul 24 19:17:58.987736 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.987580    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c47452-8e50-47db-9238-1c82b1c79cb8-plugins-config-volume\") pod \"56c47452-8e50-47db-9238-1c82b1c79cb8\" (UID: \"56c47452-8e50-47db-9238-1c82b1c79cb8\") "
Jul 24 19:17:58.987736 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.987659    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mndtx\" (UniqueName: \"kubernetes.io/projected/56c47452-8e50-47db-9238-1c82b1c79cb8-kube-api-access-mndtx\") pod \"56c47452-8e50-47db-9238-1c82b1c79cb8\" (UID: \"56c47452-8e50-47db-9238-1c82b1c79cb8\") "
Jul 24 19:17:58.987860 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.987836    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54537b19-47f5-4561-9450-48e1a7e96f42-maas-api-tls\") pod \"54537b19-47f5-4561-9450-48e1a7e96f42\" (UID: \"54537b19-47f5-4561-9450-48e1a7e96f42\") "
Jul 24 19:17:58.987918 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.987868    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48j64\" (UniqueName: \"kubernetes.io/projected/00e54db6-c033-4ffd-887a-ce0e325febb5-kube-api-access-48j64\") pod \"00e54db6-c033-4ffd-887a-ce0e325febb5\" (UID: \"00e54db6-c033-4ffd-887a-ce0e325febb5\") "
Jul 24 19:17:58.987918 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.987891    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e54db6-c033-4ffd-887a-ce0e325febb5-plugins-config-volume\") pod \"00e54db6-c033-4ffd-887a-ce0e325febb5\" (UID: \"00e54db6-c033-4ffd-887a-ce0e325febb5\") "
Jul 24 19:17:58.988020 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.987921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r24p\" (UniqueName: \"kubernetes.io/projected/54537b19-47f5-4561-9450-48e1a7e96f42-kube-api-access-6r24p\") pod \"54537b19-47f5-4561-9450-48e1a7e96f42\" (UID: \"54537b19-47f5-4561-9450-48e1a7e96f42\") "
Jul 24 19:17:58.988020 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.987941    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54537b19-47f5-4561-9450-48e1a7e96f42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "54537b19-47f5-4561-9450-48e1a7e96f42" (UID: "54537b19-47f5-4561-9450-48e1a7e96f42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:17:58.988133 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.988020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c47452-8e50-47db-9238-1c82b1c79cb8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56c47452-8e50-47db-9238-1c82b1c79cb8" (UID: "56c47452-8e50-47db-9238-1c82b1c79cb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:17:58.988286 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.988263    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54537b19-47f5-4561-9450-48e1a7e96f42-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:17:58.988430 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.988289    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c47452-8e50-47db-9238-1c82b1c79cb8-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:17:58.988430 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.988337    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e54db6-c033-4ffd-887a-ce0e325febb5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00e54db6-c033-4ffd-887a-ce0e325febb5" (UID: "00e54db6-c033-4ffd-887a-ce0e325febb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:17:58.989817 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.989793    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c47452-8e50-47db-9238-1c82b1c79cb8-kube-api-access-mndtx" (OuterVolumeSpecName: "kube-api-access-mndtx") pod "56c47452-8e50-47db-9238-1c82b1c79cb8" (UID: "56c47452-8e50-47db-9238-1c82b1c79cb8"). InnerVolumeSpecName "kube-api-access-mndtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:17:58.989928 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.989909    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e54db6-c033-4ffd-887a-ce0e325febb5-kube-api-access-48j64" (OuterVolumeSpecName: "kube-api-access-48j64") pod "00e54db6-c033-4ffd-887a-ce0e325febb5" (UID: "00e54db6-c033-4ffd-887a-ce0e325febb5"). InnerVolumeSpecName "kube-api-access-48j64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:17:58.989928 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.989911    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54537b19-47f5-4561-9450-48e1a7e96f42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "54537b19-47f5-4561-9450-48e1a7e96f42" (UID: "54537b19-47f5-4561-9450-48e1a7e96f42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:17:58.990083 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:58.990066    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54537b19-47f5-4561-9450-48e1a7e96f42-kube-api-access-6r24p" (OuterVolumeSpecName: "kube-api-access-6r24p") pod "54537b19-47f5-4561-9450-48e1a7e96f42" (UID: "54537b19-47f5-4561-9450-48e1a7e96f42"). InnerVolumeSpecName "kube-api-access-6r24p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:17:59.089735 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.089706    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mndtx\" (UniqueName: \"kubernetes.io/projected/56c47452-8e50-47db-9238-1c82b1c79cb8-kube-api-access-mndtx\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:17:59.089735 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.089732    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54537b19-47f5-4561-9450-48e1a7e96f42-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:17:59.089899 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.089746    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48j64\" (UniqueName: \"kubernetes.io/projected/00e54db6-c033-4ffd-887a-ce0e325febb5-kube-api-access-48j64\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:17:59.089899 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.089758    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e54db6-c033-4ffd-887a-ce0e325febb5-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:17:59.089899 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.089769    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r24p\" (UniqueName: \"kubernetes.io/projected/54537b19-47f5-4561-9450-48e1a7e96f42-kube-api-access-6r24p\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:17:59.726606 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.726571    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"
Jul 24 19:17:59.726606 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.726583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6" event={"ID":"00e54db6-c033-4ffd-887a-ce0e325febb5","Type":"ContainerDied","Data":"df877c36a5736104fa2f7a0c8cd6ace9f1855ee0fffa389237ec07a679a6cbf7"}
Jul 24 19:17:59.727116 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.726652    2580 scope.go:117] "RemoveContainer" containerID="0e3ee32104a44a6e72de4f5af2d2bcc2d6106268ddfca2fceecbeb69e3f0f362"
Jul 24 19:17:59.728094 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.728027    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"
Jul 24 19:17:59.728094 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.728050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7" event={"ID":"56c47452-8e50-47db-9238-1c82b1c79cb8","Type":"ContainerDied","Data":"13fc6aeb5ea097931671cb0294ac6bf37bc03fc6427ed924b68fa363ef877dbe"}
Jul 24 19:17:59.729544 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.729524    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7" event={"ID":"54537b19-47f5-4561-9450-48e1a7e96f42","Type":"ContainerDied","Data":"29766fa6e79ae1684d9b0fa667e2b507afdcd8ca85b3331a5fb3e533ab29af74"}
Jul 24 19:17:59.729544 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.729544    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"
Jul 24 19:17:59.738488 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.738468    2580 scope.go:117] "RemoveContainer" containerID="f78aa909d47d2faf23920cb37b5b36770419e45ad45b3cb57fe6d5a197c0e6da"
Jul 24 19:17:59.746090 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.746073    2580 scope.go:117] "RemoveContainer" containerID="881ec8b8b075cdb55e69dcdf69d93bc1c1586635114eb438614638c6fb71d73a"
Jul 24 19:17:59.751040 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.751019    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"]
Jul 24 19:17:59.755079 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.755061    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ee838-6b69779c-jl2r7"]
Jul 24 19:17:59.765220 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.765199    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"]
Jul 24 19:17:59.769025 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.769006    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ee838-588487cb76-krkg6"]
Jul 24 19:17:59.778720 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.778698    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"]
Jul 24 19:17:59.783223 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:17:59.783162    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ee838-76c5568d6d-9qzl7"]
Jul 24 19:18:01.203566 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:01.203537    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00e54db6-c033-4ffd-887a-ce0e325febb5" path="/var/lib/kubelet/pods/00e54db6-c033-4ffd-887a-ce0e325febb5/volumes"
Jul 24 19:18:01.203929 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:01.203916    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54537b19-47f5-4561-9450-48e1a7e96f42" path="/var/lib/kubelet/pods/54537b19-47f5-4561-9450-48e1a7e96f42/volumes"
Jul 24 19:18:01.204261 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:01.204249    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56c47452-8e50-47db-9238-1c82b1c79cb8" path="/var/lib/kubelet/pods/56c47452-8e50-47db-9238-1c82b1c79cb8/volumes"
Jul 24 19:18:20.648421 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:20.648367    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"]
Jul 24 19:18:20.648996 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:20.648603    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk" podUID="558c30f7-d5cd-479c-b3a3-855cc388f392" containerName="maas-api" containerID="cri-o://71e6e18301392bc3610db768e7ff385d7bcca1ac082cf7246f688c9bf7a2ca04" gracePeriod=30
Jul 24 19:18:20.714891 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:20.714856    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"]
Jul 24 19:18:20.715186 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:20.715138    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5" podUID="59430d9c-234d-47f9-abbd-82f9c4ec1ba0" containerName="payload-processing" containerID="cri-o://4e49a9f391a89fd4e84f28cd6cb55916249ad336da94fe8bca5ecf68bd5f5fce" gracePeriod=30
Jul 24 19:18:20.721329 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:20.721163    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"]
Jul 24 19:18:20.721463 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:20.721404    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh" podUID="e472270b-16f0-4c5a-a1cb-954e367987d0" containerName="payload-pre-processing" containerID="cri-o://75a59ee8883b70ac5807f379d14e0079d0905019b2d1edf65f1571c9f95cb352" gracePeriod=30
Jul 24 19:18:20.833943 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:20.833908    2580 generic.go:356] "Generic (PLEG): container finished" podID="59430d9c-234d-47f9-abbd-82f9c4ec1ba0" containerID="4e49a9f391a89fd4e84f28cd6cb55916249ad336da94fe8bca5ecf68bd5f5fce" exitCode=0
Jul 24 19:18:20.834070 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:20.833953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5" event={"ID":"59430d9c-234d-47f9-abbd-82f9c4ec1ba0","Type":"ContainerDied","Data":"4e49a9f391a89fd4e84f28cd6cb55916249ad336da94fe8bca5ecf68bd5f5fce"}
Jul 24 19:18:21.011506 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.011479    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"
Jul 24 19:18:21.014597 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.014576    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"
Jul 24 19:18:21.075059 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.075024    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e472270b-16f0-4c5a-a1cb-954e367987d0-plugins-config-volume\") pod \"e472270b-16f0-4c5a-a1cb-954e367987d0\" (UID: \"e472270b-16f0-4c5a-a1cb-954e367987d0\") "
Jul 24 19:18:21.075220 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.075114    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q2kj\" (UniqueName: \"kubernetes.io/projected/59430d9c-234d-47f9-abbd-82f9c4ec1ba0-kube-api-access-5q2kj\") pod \"59430d9c-234d-47f9-abbd-82f9c4ec1ba0\" (UID: \"59430d9c-234d-47f9-abbd-82f9c4ec1ba0\") "
Jul 24 19:18:21.075220 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.075149    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkgq5\" (UniqueName: \"kubernetes.io/projected/e472270b-16f0-4c5a-a1cb-954e367987d0-kube-api-access-dkgq5\") pod \"e472270b-16f0-4c5a-a1cb-954e367987d0\" (UID: \"e472270b-16f0-4c5a-a1cb-954e367987d0\") "
Jul 24 19:18:21.075220 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.075203    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59430d9c-234d-47f9-abbd-82f9c4ec1ba0-plugins-config-volume\") pod \"59430d9c-234d-47f9-abbd-82f9c4ec1ba0\" (UID: \"59430d9c-234d-47f9-abbd-82f9c4ec1ba0\") "
Jul 24 19:18:21.075662 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.075474    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e472270b-16f0-4c5a-a1cb-954e367987d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e472270b-16f0-4c5a-a1cb-954e367987d0" (UID: "e472270b-16f0-4c5a-a1cb-954e367987d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:18:21.075786 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.075699    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59430d9c-234d-47f9-abbd-82f9c4ec1ba0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59430d9c-234d-47f9-abbd-82f9c4ec1ba0" (UID: "59430d9c-234d-47f9-abbd-82f9c4ec1ba0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:18:21.077362 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.077338    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59430d9c-234d-47f9-abbd-82f9c4ec1ba0-kube-api-access-5q2kj" (OuterVolumeSpecName: "kube-api-access-5q2kj") pod "59430d9c-234d-47f9-abbd-82f9c4ec1ba0" (UID: "59430d9c-234d-47f9-abbd-82f9c4ec1ba0"). InnerVolumeSpecName "kube-api-access-5q2kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:18:21.077499 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.077410    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e472270b-16f0-4c5a-a1cb-954e367987d0-kube-api-access-dkgq5" (OuterVolumeSpecName: "kube-api-access-dkgq5") pod "e472270b-16f0-4c5a-a1cb-954e367987d0" (UID: "e472270b-16f0-4c5a-a1cb-954e367987d0"). InnerVolumeSpecName "kube-api-access-dkgq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:18:21.092959 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.092940    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:18:21.176724 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.176653    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/558c30f7-d5cd-479c-b3a3-855cc388f392-maas-api-tls\") pod \"558c30f7-d5cd-479c-b3a3-855cc388f392\" (UID: \"558c30f7-d5cd-479c-b3a3-855cc388f392\") "
Jul 24 19:18:21.176860 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.176748    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/558c30f7-d5cd-479c-b3a3-855cc388f392-openshift-service-ca\") pod \"558c30f7-d5cd-479c-b3a3-855cc388f392\" (UID: \"558c30f7-d5cd-479c-b3a3-855cc388f392\") "
Jul 24 19:18:21.176860 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.176779    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd5jp\" (UniqueName: \"kubernetes.io/projected/558c30f7-d5cd-479c-b3a3-855cc388f392-kube-api-access-vd5jp\") pod \"558c30f7-d5cd-479c-b3a3-855cc388f392\" (UID: \"558c30f7-d5cd-479c-b3a3-855cc388f392\") "
Jul 24 19:18:21.177037 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.177019    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59430d9c-234d-47f9-abbd-82f9c4ec1ba0-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:18:21.177095 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.177043    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e472270b-16f0-4c5a-a1cb-954e367987d0-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:18:21.177095 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.177059    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q2kj\" (UniqueName: \"kubernetes.io/projected/59430d9c-234d-47f9-abbd-82f9c4ec1ba0-kube-api-access-5q2kj\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:18:21.177095 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.177040    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558c30f7-d5cd-479c-b3a3-855cc388f392-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "558c30f7-d5cd-479c-b3a3-855cc388f392" (UID: "558c30f7-d5cd-479c-b3a3-855cc388f392"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:18:21.177095 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.177073    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkgq5\" (UniqueName: \"kubernetes.io/projected/e472270b-16f0-4c5a-a1cb-954e367987d0-kube-api-access-dkgq5\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:18:21.178691 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.178669    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/558c30f7-d5cd-479c-b3a3-855cc388f392-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "558c30f7-d5cd-479c-b3a3-855cc388f392" (UID: "558c30f7-d5cd-479c-b3a3-855cc388f392"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:18:21.178756 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.178696    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558c30f7-d5cd-479c-b3a3-855cc388f392-kube-api-access-vd5jp" (OuterVolumeSpecName: "kube-api-access-vd5jp") pod "558c30f7-d5cd-479c-b3a3-855cc388f392" (UID: "558c30f7-d5cd-479c-b3a3-855cc388f392"). InnerVolumeSpecName "kube-api-access-vd5jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:18:21.277577 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.277544    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/558c30f7-d5cd-479c-b3a3-855cc388f392-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:18:21.277678 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.277579    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd5jp\" (UniqueName: \"kubernetes.io/projected/558c30f7-d5cd-479c-b3a3-855cc388f392-kube-api-access-vd5jp\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:18:21.277678 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.277594    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/558c30f7-d5cd-479c-b3a3-855cc388f392-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:18:21.842111 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.842078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5" event={"ID":"59430d9c-234d-47f9-abbd-82f9c4ec1ba0","Type":"ContainerDied","Data":"45b36f2c05bfe71924d636752d13b223fedae68bd29171bf036412e4bf937eeb"}
Jul 24 19:18:21.842590 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.842142    2580 scope.go:117] "RemoveContainer" containerID="4e49a9f391a89fd4e84f28cd6cb55916249ad336da94fe8bca5ecf68bd5f5fce"
Jul 24 19:18:21.842590 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.842092    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"
Jul 24 19:18:21.843364 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.843307    2580 generic.go:356] "Generic (PLEG): container finished" podID="e472270b-16f0-4c5a-a1cb-954e367987d0" containerID="75a59ee8883b70ac5807f379d14e0079d0905019b2d1edf65f1571c9f95cb352" exitCode=0
Jul 24 19:18:21.843461 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.843373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh" event={"ID":"e472270b-16f0-4c5a-a1cb-954e367987d0","Type":"ContainerDied","Data":"75a59ee8883b70ac5807f379d14e0079d0905019b2d1edf65f1571c9f95cb352"}
Jul 24 19:18:21.843461 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.843398    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"
Jul 24 19:18:21.843461 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.843425    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh" event={"ID":"e472270b-16f0-4c5a-a1cb-954e367987d0","Type":"ContainerDied","Data":"db08eb57529dc1945eb12d69a40d55bcc463732a1139f5ba93ce6d4c3c8ed97a"}
Jul 24 19:18:21.844879 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.844855    2580 generic.go:356] "Generic (PLEG): container finished" podID="558c30f7-d5cd-479c-b3a3-855cc388f392" containerID="71e6e18301392bc3610db768e7ff385d7bcca1ac082cf7246f688c9bf7a2ca04" exitCode=0
Jul 24 19:18:21.844998 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.844896    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"
Jul 24 19:18:21.844998 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.844935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk" event={"ID":"558c30f7-d5cd-479c-b3a3-855cc388f392","Type":"ContainerDied","Data":"71e6e18301392bc3610db768e7ff385d7bcca1ac082cf7246f688c9bf7a2ca04"}
Jul 24 19:18:21.844998 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.844992    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk" event={"ID":"558c30f7-d5cd-479c-b3a3-855cc388f392","Type":"ContainerDied","Data":"0a9d7c58f92ea2df1f44e58297a76bf8c8431142703c2ad4c333a833073ebe7e"}
Jul 24 19:18:21.851669 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.851651    2580 scope.go:117] "RemoveContainer" containerID="75a59ee8883b70ac5807f379d14e0079d0905019b2d1edf65f1571c9f95cb352"
Jul 24 19:18:21.859148 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.859131    2580 scope.go:117] "RemoveContainer" containerID="75a59ee8883b70ac5807f379d14e0079d0905019b2d1edf65f1571c9f95cb352"
Jul 24 19:18:21.859373 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:18:21.859356    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a59ee8883b70ac5807f379d14e0079d0905019b2d1edf65f1571c9f95cb352\": container with ID starting with 75a59ee8883b70ac5807f379d14e0079d0905019b2d1edf65f1571c9f95cb352 not found: ID does not exist" containerID="75a59ee8883b70ac5807f379d14e0079d0905019b2d1edf65f1571c9f95cb352"
Jul 24 19:18:21.859454 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.859393    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a59ee8883b70ac5807f379d14e0079d0905019b2d1edf65f1571c9f95cb352"} err="failed to get container status \"75a59ee8883b70ac5807f379d14e0079d0905019b2d1edf65f1571c9f95cb352\": rpc error: code = NotFound desc = could not find container \"75a59ee8883b70ac5807f379d14e0079d0905019b2d1edf65f1571c9f95cb352\": container with ID starting with 75a59ee8883b70ac5807f379d14e0079d0905019b2d1edf65f1571c9f95cb352 not found: ID does not exist"
Jul 24 19:18:21.859454 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.859410    2580 scope.go:117] "RemoveContainer" containerID="71e6e18301392bc3610db768e7ff385d7bcca1ac082cf7246f688c9bf7a2ca04"
Jul 24 19:18:21.864636 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.864616    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"]
Jul 24 19:18:21.867260 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.867242    2580 scope.go:117] "RemoveContainer" containerID="71e6e18301392bc3610db768e7ff385d7bcca1ac082cf7246f688c9bf7a2ca04"
Jul 24 19:18:21.867544 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:18:21.867515    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e6e18301392bc3610db768e7ff385d7bcca1ac082cf7246f688c9bf7a2ca04\": container with ID starting with 71e6e18301392bc3610db768e7ff385d7bcca1ac082cf7246f688c9bf7a2ca04 not found: ID does not exist" containerID="71e6e18301392bc3610db768e7ff385d7bcca1ac082cf7246f688c9bf7a2ca04"
Jul 24 19:18:21.867652 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.867558    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e6e18301392bc3610db768e7ff385d7bcca1ac082cf7246f688c9bf7a2ca04"} err="failed to get container status \"71e6e18301392bc3610db768e7ff385d7bcca1ac082cf7246f688c9bf7a2ca04\": rpc error: code = NotFound desc = could not find container \"71e6e18301392bc3610db768e7ff385d7bcca1ac082cf7246f688c9bf7a2ca04\": container with ID starting with 71e6e18301392bc3610db768e7ff385d7bcca1ac082cf7246f688c9bf7a2ca04 not found: ID does not exist"
Jul 24 19:18:21.870437 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.870415    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-31c4fa-85bd8d9d64-8s8v5"]
Jul 24 19:18:21.879743 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.879723    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"]
Jul 24 19:18:21.883899 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.883882    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-31c4fa-7b57f8578d-4zmlh"]
Jul 24 19:18:21.895000 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.894979    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"]
Jul 24 19:18:21.900981 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:21.900962    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-31c4fa-85d597fdd7-4pfjk"]
Jul 24 19:18:23.197066 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:23.197036    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="558c30f7-d5cd-479c-b3a3-855cc388f392" path="/var/lib/kubelet/pods/558c30f7-d5cd-479c-b3a3-855cc388f392/volumes"
Jul 24 19:18:23.197453 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:23.197405    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59430d9c-234d-47f9-abbd-82f9c4ec1ba0" path="/var/lib/kubelet/pods/59430d9c-234d-47f9-abbd-82f9c4ec1ba0/volumes"
Jul 24 19:18:23.197727 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:23.197714    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e472270b-16f0-4c5a-a1cb-954e367987d0" path="/var/lib/kubelet/pods/e472270b-16f0-4c5a-a1cb-954e367987d0/volumes"
Jul 24 19:18:36.036747 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.036714    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"]
Jul 24 19:18:36.037409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037158    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e54db6-c033-4ffd-887a-ce0e325febb5" containerName="payload-pre-processing"
Jul 24 19:18:36.037409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037175    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e54db6-c033-4ffd-887a-ce0e325febb5" containerName="payload-pre-processing"
Jul 24 19:18:36.037409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037203    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c47452-8e50-47db-9238-1c82b1c79cb8" containerName="payload-processing"
Jul 24 19:18:36.037409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037212    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c47452-8e50-47db-9238-1c82b1c79cb8" containerName="payload-processing"
Jul 24 19:18:36.037409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037222    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54537b19-47f5-4561-9450-48e1a7e96f42" containerName="maas-api"
Jul 24 19:18:36.037409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037232    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="54537b19-47f5-4561-9450-48e1a7e96f42" containerName="maas-api"
Jul 24 19:18:36.037409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037244    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558c30f7-d5cd-479c-b3a3-855cc388f392" containerName="maas-api"
Jul 24 19:18:36.037409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037251    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="558c30f7-d5cd-479c-b3a3-855cc388f392" containerName="maas-api"
Jul 24 19:18:36.037409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037265    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59430d9c-234d-47f9-abbd-82f9c4ec1ba0" containerName="payload-processing"
Jul 24 19:18:36.037409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037273    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="59430d9c-234d-47f9-abbd-82f9c4ec1ba0" containerName="payload-processing"
Jul 24 19:18:36.037409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037290    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e472270b-16f0-4c5a-a1cb-954e367987d0" containerName="payload-pre-processing"
Jul 24 19:18:36.037409 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037298    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e472270b-16f0-4c5a-a1cb-954e367987d0" containerName="payload-pre-processing"
Jul 24 19:18:36.037885 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037418    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="54537b19-47f5-4561-9450-48e1a7e96f42" containerName="maas-api"
Jul 24 19:18:36.037885 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037434    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="59430d9c-234d-47f9-abbd-82f9c4ec1ba0" containerName="payload-processing"
Jul 24 19:18:36.037885 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037444    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e54db6-c033-4ffd-887a-ce0e325febb5" containerName="payload-pre-processing"
Jul 24 19:18:36.037885 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037455    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e472270b-16f0-4c5a-a1cb-954e367987d0" containerName="payload-pre-processing"
Jul 24 19:18:36.037885 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037467    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="56c47452-8e50-47db-9238-1c82b1c79cb8" containerName="payload-processing"
Jul 24 19:18:36.037885 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.037475    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="558c30f7-d5cd-479c-b3a3-855cc388f392" containerName="maas-api"
Jul 24 19:18:36.041972 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.041956    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:18:36.044865 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.044841    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5e9947-serving-cert\""
Jul 24 19:18:36.068711 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.068678    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"]
Jul 24 19:18:36.115249 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.115217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jrvz\" (UniqueName: \"kubernetes.io/projected/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-kube-api-access-2jrvz\") pod \"maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm\" (UID: \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:18:36.115370 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.115337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-maas-api-tls\") pod \"maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm\" (UID: \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:18:36.115370 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.115357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-openshift-service-ca\") pod \"maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm\" (UID: \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:18:36.216256 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.216228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-maas-api-tls\") pod \"maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm\" (UID: \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:18:36.216420 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.216262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-openshift-service-ca\") pod \"maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm\" (UID: \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:18:36.216420 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.216295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jrvz\" (UniqueName: \"kubernetes.io/projected/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-kube-api-access-2jrvz\") pod \"maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm\" (UID: \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:18:36.216522 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:18:36.216417    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-serving-cert: secret "maas-api-e2e-api-a-5e9947-serving-cert" not found
Jul 24 19:18:36.216522 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:18:36.216489    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-maas-api-tls podName:b1e614b8-fa6f-4640-8744-ba1d04cea4b2 nodeName:}" failed. No retries permitted until 2026-07-24 19:18:36.716469533 +0000 UTC m=+2316.092128812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-maas-api-tls") pod "maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm" (UID: "b1e614b8-fa6f-4640-8744-ba1d04cea4b2") : secret "maas-api-e2e-api-a-5e9947-serving-cert" not found
Jul 24 19:18:36.216882 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.216864    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-openshift-service-ca\") pod \"maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm\" (UID: \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:18:36.226162 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.226138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jrvz\" (UniqueName: \"kubernetes.io/projected/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-kube-api-access-2jrvz\") pod \"maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm\" (UID: \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:18:36.688179 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.688144    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"]
Jul 24 19:18:36.691873 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.691854    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"
Jul 24 19:18:36.694401 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.694361    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5e9947\""
Jul 24 19:18:36.701616 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.701598    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"]
Jul 24 19:18:36.718275 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.718245    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"]
Jul 24 19:18:36.720755 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.720733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-maas-api-tls\") pod \"maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm\" (UID: \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:18:36.721656 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.721636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"
Jul 24 19:18:36.723180 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.723155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-maas-api-tls\") pod \"maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm\" (UID: \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:18:36.723745 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.723728    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5e9947-dockercfg-4wmh6\""
Jul 24 19:18:36.730826 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.730802    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"]
Jul 24 19:18:36.822106 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.822075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44e535-42d7-426e-b7da-b38de1adacf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq\" (UID: \"9f44e535-42d7-426e-b7da-b38de1adacf2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"
Jul 24 19:18:36.822259 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.822127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987e6cc1-1149-4a69-9bc7-69a216da89da-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f\" (UID: \"987e6cc1-1149-4a69-9bc7-69a216da89da\") " pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"
Jul 24 19:18:36.822259 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.822220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv6wb\" (UniqueName: \"kubernetes.io/projected/9f44e535-42d7-426e-b7da-b38de1adacf2-kube-api-access-nv6wb\") pod \"payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq\" (UID: \"9f44e535-42d7-426e-b7da-b38de1adacf2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"
Jul 24 19:18:36.822333 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.822289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zr6k\" (UniqueName: \"kubernetes.io/projected/987e6cc1-1149-4a69-9bc7-69a216da89da-kube-api-access-4zr6k\") pod \"payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f\" (UID: \"987e6cc1-1149-4a69-9bc7-69a216da89da\") " pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"
Jul 24 19:18:36.923143 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.923104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44e535-42d7-426e-b7da-b38de1adacf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq\" (UID: \"9f44e535-42d7-426e-b7da-b38de1adacf2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"
Jul 24 19:18:36.923295 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.923203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987e6cc1-1149-4a69-9bc7-69a216da89da-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f\" (UID: \"987e6cc1-1149-4a69-9bc7-69a216da89da\") " pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"
Jul 24 19:18:36.923295 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.923248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6wb\" (UniqueName: \"kubernetes.io/projected/9f44e535-42d7-426e-b7da-b38de1adacf2-kube-api-access-nv6wb\") pod \"payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq\" (UID: \"9f44e535-42d7-426e-b7da-b38de1adacf2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"
Jul 24 19:18:36.923418 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.923316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zr6k\" (UniqueName: \"kubernetes.io/projected/987e6cc1-1149-4a69-9bc7-69a216da89da-kube-api-access-4zr6k\") pod \"payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f\" (UID: \"987e6cc1-1149-4a69-9bc7-69a216da89da\") " pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"
Jul 24 19:18:36.924066 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.924040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987e6cc1-1149-4a69-9bc7-69a216da89da-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f\" (UID: \"987e6cc1-1149-4a69-9bc7-69a216da89da\") " pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"
Jul 24 19:18:36.924152 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.924050    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44e535-42d7-426e-b7da-b38de1adacf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq\" (UID: \"9f44e535-42d7-426e-b7da-b38de1adacf2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"
Jul 24 19:18:36.931564 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.931542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv6wb\" (UniqueName: \"kubernetes.io/projected/9f44e535-42d7-426e-b7da-b38de1adacf2-kube-api-access-nv6wb\") pod \"payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq\" (UID: \"9f44e535-42d7-426e-b7da-b38de1adacf2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"
Jul 24 19:18:36.931683 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.931665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zr6k\" (UniqueName: \"kubernetes.io/projected/987e6cc1-1149-4a69-9bc7-69a216da89da-kube-api-access-4zr6k\") pod \"payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f\" (UID: \"987e6cc1-1149-4a69-9bc7-69a216da89da\") " pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"
Jul 24 19:18:36.952488 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:36.952433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:18:37.003941 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.003910    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"
Jul 24 19:18:37.041885 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.041818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"
Jul 24 19:18:37.099504 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.099299    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"]
Jul 24 19:18:37.106235 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:18:37.104464    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e614b8_fa6f_4640_8744_ba1d04cea4b2.slice/crio-919a117ba5e76096c188fbd19cc74be0ff0a6cdb2db6900ac5c0743478818a26 WatchSource:0}: Error finding container 919a117ba5e76096c188fbd19cc74be0ff0a6cdb2db6900ac5c0743478818a26: Status 404 returned error can't find the container with id 919a117ba5e76096c188fbd19cc74be0ff0a6cdb2db6900ac5c0743478818a26
Jul 24 19:18:37.107133 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.107103    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 19:18:37.164950 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.164764    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"]
Jul 24 19:18:37.167935 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:18:37.167903    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f44e535_42d7_426e_b7da_b38de1adacf2.slice/crio-4941b035247e377671a998884c9ce31140338e1e35a1d81dae716d7c2fef6cea WatchSource:0}: Error finding container 4941b035247e377671a998884c9ce31140338e1e35a1d81dae716d7c2fef6cea: Status 404 returned error can't find the container with id 4941b035247e377671a998884c9ce31140338e1e35a1d81dae716d7c2fef6cea
Jul 24 19:18:37.212824 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.212800    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"]
Jul 24 19:18:37.216429 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:18:37.216401    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987e6cc1_1149_4a69_9bc7_69a216da89da.slice/crio-6728aa0534bc5b7f86cbe372b36096d3d69c91025355ea19ef929282668bdba4 WatchSource:0}: Error finding container 6728aa0534bc5b7f86cbe372b36096d3d69c91025355ea19ef929282668bdba4: Status 404 returned error can't find the container with id 6728aa0534bc5b7f86cbe372b36096d3d69c91025355ea19ef929282668bdba4
Jul 24 19:18:37.926119 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.926031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f" event={"ID":"987e6cc1-1149-4a69-9bc7-69a216da89da","Type":"ContainerStarted","Data":"800a784c684d6a2f1cccc9b3084b35c599ab83b3a0c192f91439ef484a5f2d20"}
Jul 24 19:18:37.926119 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.926090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f" event={"ID":"987e6cc1-1149-4a69-9bc7-69a216da89da","Type":"ContainerStarted","Data":"6728aa0534bc5b7f86cbe372b36096d3d69c91025355ea19ef929282668bdba4"}
Jul 24 19:18:37.926337 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.926203    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"
Jul 24 19:18:37.927664 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.927632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm" event={"ID":"b1e614b8-fa6f-4640-8744-ba1d04cea4b2","Type":"ContainerStarted","Data":"5523d7b87dd9eb82c73aa9596ce16b790bbdfdcc3befffcd67a769bdfae4db1e"}
Jul 24 19:18:37.927789 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.927682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm" event={"ID":"b1e614b8-fa6f-4640-8744-ba1d04cea4b2","Type":"ContainerStarted","Data":"919a117ba5e76096c188fbd19cc74be0ff0a6cdb2db6900ac5c0743478818a26"}
Jul 24 19:18:37.927789 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.927711    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:18:37.929229 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.929203    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq" event={"ID":"9f44e535-42d7-426e-b7da-b38de1adacf2","Type":"ContainerStarted","Data":"c2d37e5923200342aeb405066afb222ad56d367423a7856ac421b31cc1a1ac1a"}
Jul 24 19:18:37.929353 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.929249    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq" event={"ID":"9f44e535-42d7-426e-b7da-b38de1adacf2","Type":"ContainerStarted","Data":"4941b035247e377671a998884c9ce31140338e1e35a1d81dae716d7c2fef6cea"}
Jul 24 19:18:37.929353 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.929300    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"
Jul 24 19:18:37.942750 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.942706    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f" podStartSLOduration=1.94268857 podStartE2EDuration="1.94268857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:18:37.94146115 +0000 UTC m=+2317.317120445" watchObservedRunningTime="2026-07-24 19:18:37.94268857 +0000 UTC m=+2317.318347849"
Jul 24 19:18:37.959150 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.959097    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq" podStartSLOduration=1.9590798999999999 podStartE2EDuration="1.9590799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:18:37.956534236 +0000 UTC m=+2317.332193537" watchObservedRunningTime="2026-07-24 19:18:37.9590799 +0000 UTC m=+2317.334739198"
Jul 24 19:18:37.974912 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:37.974865    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm" podStartSLOduration=1.500955611 podStartE2EDuration="1.974847173s" totalImagesPullingTime="473.891562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:18:37.973296048 +0000 UTC m=+2317.348955344" watchObservedRunningTime="2026-07-24 19:18:37.974847173 +0000 UTC m=+2317.350506473"
Jul 24 19:18:43.893706 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:43.893670    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"]
Jul 24 19:18:43.897742 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:43.897721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:18:43.899734 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:43.899704    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a90216-serving-cert\""
Jul 24 19:18:43.908228 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:43.908200    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"]
Jul 24 19:18:43.940499 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:43.940478    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:18:43.986707 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:43.986669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mthb\" (UniqueName: \"kubernetes.io/projected/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-kube-api-access-5mthb\") pod \"maas-api-e2e-api-b-a90216-c689f9875-wtf9d\" (UID: \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:18:43.986852 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:43.986766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-openshift-service-ca\") pod \"maas-api-e2e-api-b-a90216-c689f9875-wtf9d\" (UID: \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:18:43.986852 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:43.986809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-maas-api-tls\") pod \"maas-api-e2e-api-b-a90216-c689f9875-wtf9d\" (UID: \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:18:44.087460 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.087433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mthb\" (UniqueName: \"kubernetes.io/projected/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-kube-api-access-5mthb\") pod \"maas-api-e2e-api-b-a90216-c689f9875-wtf9d\" (UID: \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:18:44.087620 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.087510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-openshift-service-ca\") pod \"maas-api-e2e-api-b-a90216-c689f9875-wtf9d\" (UID: \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:18:44.087620 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.087545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-maas-api-tls\") pod \"maas-api-e2e-api-b-a90216-c689f9875-wtf9d\" (UID: \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:18:44.088509 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.088486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-openshift-service-ca\") pod \"maas-api-e2e-api-b-a90216-c689f9875-wtf9d\" (UID: \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:18:44.090124 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.090102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-maas-api-tls\") pod \"maas-api-e2e-api-b-a90216-c689f9875-wtf9d\" (UID: \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:18:44.101051 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.101028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mthb\" (UniqueName: \"kubernetes.io/projected/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-kube-api-access-5mthb\") pod \"maas-api-e2e-api-b-a90216-c689f9875-wtf9d\" (UID: \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:18:44.208762 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.208683    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:18:44.342945 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.342881    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"]
Jul 24 19:18:44.345848 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:18:44.345821    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb2bad8_8c3d_4705_a6d3_6a674bc6acf9.slice/crio-bf86765e27acbcc5673bee94d973347ce3288602187f7df890acee18b8090583 WatchSource:0}: Error finding container bf86765e27acbcc5673bee94d973347ce3288602187f7df890acee18b8090583: Status 404 returned error can't find the container with id bf86765e27acbcc5673bee94d973347ce3288602187f7df890acee18b8090583
Jul 24 19:18:44.684469 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.684439    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"]
Jul 24 19:18:44.688146 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.688126    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"
Jul 24 19:18:44.691580 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.691555    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a90216-dockercfg-bgjsz\""
Jul 24 19:18:44.691699 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.691649    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a90216\""
Jul 24 19:18:44.699646 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.699623    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"]
Jul 24 19:18:44.775606 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.775579    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"]
Jul 24 19:18:44.779009 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.778993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"
Jul 24 19:18:44.789484 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.789460    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"]
Jul 24 19:18:44.795166 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.795144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3ead70-c149-49e2-97b1-b00e5536a456-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2\" (UID: \"9f3ead70-c149-49e2-97b1-b00e5536a456\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"
Jul 24 19:18:44.795260 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.795186    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6p2k\" (UniqueName: \"kubernetes.io/projected/9f3ead70-c149-49e2-97b1-b00e5536a456-kube-api-access-v6p2k\") pod \"payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2\" (UID: \"9f3ead70-c149-49e2-97b1-b00e5536a456\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"
Jul 24 19:18:44.895924 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.895890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025f68b6-59bb-470b-90eb-541b8c585210-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr\" (UID: \"025f68b6-59bb-470b-90eb-541b8c585210\") " pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"
Jul 24 19:18:44.896403 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.896037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3ead70-c149-49e2-97b1-b00e5536a456-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2\" (UID: \"9f3ead70-c149-49e2-97b1-b00e5536a456\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"
Jul 24 19:18:44.896403 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.896107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6p2k\" (UniqueName: \"kubernetes.io/projected/9f3ead70-c149-49e2-97b1-b00e5536a456-kube-api-access-v6p2k\") pod \"payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2\" (UID: \"9f3ead70-c149-49e2-97b1-b00e5536a456\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"
Jul 24 19:18:44.896403 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.896165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdp6\" (UniqueName: \"kubernetes.io/projected/025f68b6-59bb-470b-90eb-541b8c585210-kube-api-access-gjdp6\") pod \"payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr\" (UID: \"025f68b6-59bb-470b-90eb-541b8c585210\") " pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"
Jul 24 19:18:44.896743 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.896724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3ead70-c149-49e2-97b1-b00e5536a456-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2\" (UID: \"9f3ead70-c149-49e2-97b1-b00e5536a456\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"
Jul 24 19:18:44.905197 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.905174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6p2k\" (UniqueName: \"kubernetes.io/projected/9f3ead70-c149-49e2-97b1-b00e5536a456-kube-api-access-v6p2k\") pod \"payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2\" (UID: \"9f3ead70-c149-49e2-97b1-b00e5536a456\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"
Jul 24 19:18:44.967802 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.967720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d" event={"ID":"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9","Type":"ContainerStarted","Data":"4d355e6121c3a3990288fa723c2d7ed86161869d9ed8967ca7a55b25c55a46bd"}
Jul 24 19:18:44.967802 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.967769    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d" event={"ID":"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9","Type":"ContainerStarted","Data":"bf86765e27acbcc5673bee94d973347ce3288602187f7df890acee18b8090583"}
Jul 24 19:18:44.968023 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.967847    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:18:44.988546 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.988498    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d" podStartSLOduration=1.641970961 podStartE2EDuration="1.988485606s" totalImagesPullingTime="346.514645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:18:44.986241174 +0000 UTC m=+2324.361900480" watchObservedRunningTime="2026-07-24 19:18:44.988485606 +0000 UTC m=+2324.364144902"
Jul 24 19:18:44.996766 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.996739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdp6\" (UniqueName: \"kubernetes.io/projected/025f68b6-59bb-470b-90eb-541b8c585210-kube-api-access-gjdp6\") pod \"payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr\" (UID: \"025f68b6-59bb-470b-90eb-541b8c585210\") " pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"
Jul 24 19:18:44.996896 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.996778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025f68b6-59bb-470b-90eb-541b8c585210-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr\" (UID: \"025f68b6-59bb-470b-90eb-541b8c585210\") " pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"
Jul 24 19:18:44.997456 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:44.997435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025f68b6-59bb-470b-90eb-541b8c585210-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr\" (UID: \"025f68b6-59bb-470b-90eb-541b8c585210\") " pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"
Jul 24 19:18:45.000173 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:45.000155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"
Jul 24 19:18:45.004165 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:45.004147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdp6\" (UniqueName: \"kubernetes.io/projected/025f68b6-59bb-470b-90eb-541b8c585210-kube-api-access-gjdp6\") pod \"payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr\" (UID: \"025f68b6-59bb-470b-90eb-541b8c585210\") " pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"
Jul 24 19:18:45.089742 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:45.089706    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"
Jul 24 19:18:45.135885 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:45.135850    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"]
Jul 24 19:18:45.137683 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:18:45.137661    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3ead70_c149_49e2_97b1_b00e5536a456.slice/crio-772b761d7d240a366a412e12766000acdd032f61f42ea9154d415d8e7c9d057a WatchSource:0}: Error finding container 772b761d7d240a366a412e12766000acdd032f61f42ea9154d415d8e7c9d057a: Status 404 returned error can't find the container with id 772b761d7d240a366a412e12766000acdd032f61f42ea9154d415d8e7c9d057a
Jul 24 19:18:45.257591 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:45.257273    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"]
Jul 24 19:18:45.260005 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:18:45.259980    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025f68b6_59bb_470b_90eb_541b8c585210.slice/crio-05771d32d9c1aab9f4c1af5e7b564dd74728a27c8188f058a09b4e055ce61cf9 WatchSource:0}: Error finding container 05771d32d9c1aab9f4c1af5e7b564dd74728a27c8188f058a09b4e055ce61cf9: Status 404 returned error can't find the container with id 05771d32d9c1aab9f4c1af5e7b564dd74728a27c8188f058a09b4e055ce61cf9
Jul 24 19:18:45.975793 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:45.975756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2" event={"ID":"9f3ead70-c149-49e2-97b1-b00e5536a456","Type":"ContainerStarted","Data":"f20b6b8af6a30c988401db04c4a3ec5d8f801de681ab4529129e9b9665ee148e"}
Jul 24 19:18:45.976254 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:45.975827    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"
Jul 24 19:18:45.976254 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:45.975855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2" event={"ID":"9f3ead70-c149-49e2-97b1-b00e5536a456","Type":"ContainerStarted","Data":"772b761d7d240a366a412e12766000acdd032f61f42ea9154d415d8e7c9d057a"}
Jul 24 19:18:45.977613 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:45.977586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr" event={"ID":"025f68b6-59bb-470b-90eb-541b8c585210","Type":"ContainerStarted","Data":"0e35c23b16a08c04ab765c1f543cf53f4bfa344b71bb0a6282a5cfd4c969e454"}
Jul 24 19:18:45.977727 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:45.977630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr" event={"ID":"025f68b6-59bb-470b-90eb-541b8c585210","Type":"ContainerStarted","Data":"05771d32d9c1aab9f4c1af5e7b564dd74728a27c8188f058a09b4e055ce61cf9"}
Jul 24 19:18:45.977852 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:45.977825    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"
Jul 24 19:18:45.994636 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:45.994596    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2" podStartSLOduration=1.9945822309999999 podStartE2EDuration="1.994582231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:18:45.992895017 +0000 UTC m=+2325.368554312" watchObservedRunningTime="2026-07-24 19:18:45.994582231 +0000 UTC m=+2325.370241574"
Jul 24 19:18:46.011496 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:46.011451    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr" podStartSLOduration=2.011432782 podStartE2EDuration="2.011432782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:18:46.01047946 +0000 UTC m=+2325.386138756" watchObservedRunningTime="2026-07-24 19:18:46.011432782 +0000 UTC m=+2325.387092082"
Jul 24 19:18:48.937601 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:48.937550    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"
Jul 24 19:18:48.938013 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:48.937685    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"
Jul 24 19:18:50.982845 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:50.982816    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:18:56.984534 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:56.984455    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"
Jul 24 19:18:56.984534 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:18:56.984527    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"
Jul 24 19:19:01.383903 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:01.383866    2580 scope.go:117] "RemoveContainer" containerID="ae1368779b60517d866b8358d58bbafb3e28cc18b8154259ba9116fc188f4e5f"
Jul 24 19:19:11.511331 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.511292    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"]
Jul 24 19:19:11.511940 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.511552    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm" podUID="b1e614b8-fa6f-4640-8744-ba1d04cea4b2" containerName="maas-api" containerID="cri-o://5523d7b87dd9eb82c73aa9596ce16b790bbdfdcc3befffcd67a769bdfae4db1e" gracePeriod=30
Jul 24 19:19:11.566162 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.566135    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"]
Jul 24 19:19:11.566391 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.566336    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f" podUID="987e6cc1-1149-4a69-9bc7-69a216da89da" containerName="payload-processing" containerID="cri-o://800a784c684d6a2f1cccc9b3084b35c599ab83b3a0c192f91439ef484a5f2d20" gracePeriod=30
Jul 24 19:19:11.573006 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.572977    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"]
Jul 24 19:19:11.573250 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.573199    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq" podUID="9f44e535-42d7-426e-b7da-b38de1adacf2" containerName="payload-pre-processing" containerID="cri-o://c2d37e5923200342aeb405066afb222ad56d367423a7856ac421b31cc1a1ac1a" gracePeriod=30
Jul 24 19:19:11.858260 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.858230    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"
Jul 24 19:19:11.878433 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.878407    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:19:11.895882 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.895819    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"
Jul 24 19:19:11.931045 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.931022    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jrvz\" (UniqueName: \"kubernetes.io/projected/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-kube-api-access-2jrvz\") pod \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\" (UID: \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\") "
Jul 24 19:19:11.931191 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.931063    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-openshift-service-ca\") pod \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\" (UID: \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\") "
Jul 24 19:19:11.931191 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.931108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zr6k\" (UniqueName: \"kubernetes.io/projected/987e6cc1-1149-4a69-9bc7-69a216da89da-kube-api-access-4zr6k\") pod \"987e6cc1-1149-4a69-9bc7-69a216da89da\" (UID: \"987e6cc1-1149-4a69-9bc7-69a216da89da\") "
Jul 24 19:19:11.931191 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.931173    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-maas-api-tls\") pod \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\" (UID: \"b1e614b8-fa6f-4640-8744-ba1d04cea4b2\") "
Jul 24 19:19:11.931363 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.931205    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987e6cc1-1149-4a69-9bc7-69a216da89da-plugins-config-volume\") pod \"987e6cc1-1149-4a69-9bc7-69a216da89da\" (UID: \"987e6cc1-1149-4a69-9bc7-69a216da89da\") "
Jul 24 19:19:11.931521 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.931491    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b1e614b8-fa6f-4640-8744-ba1d04cea4b2" (UID: "b1e614b8-fa6f-4640-8744-ba1d04cea4b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:19:11.931885 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.931853    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987e6cc1-1149-4a69-9bc7-69a216da89da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "987e6cc1-1149-4a69-9bc7-69a216da89da" (UID: "987e6cc1-1149-4a69-9bc7-69a216da89da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:19:11.933314 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.933293    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-kube-api-access-2jrvz" (OuterVolumeSpecName: "kube-api-access-2jrvz") pod "b1e614b8-fa6f-4640-8744-ba1d04cea4b2" (UID: "b1e614b8-fa6f-4640-8744-ba1d04cea4b2"). InnerVolumeSpecName "kube-api-access-2jrvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:19:11.933419 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.933318    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b1e614b8-fa6f-4640-8744-ba1d04cea4b2" (UID: "b1e614b8-fa6f-4640-8744-ba1d04cea4b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:19:11.933419 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:11.933318    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987e6cc1-1149-4a69-9bc7-69a216da89da-kube-api-access-4zr6k" (OuterVolumeSpecName: "kube-api-access-4zr6k") pod "987e6cc1-1149-4a69-9bc7-69a216da89da" (UID: "987e6cc1-1149-4a69-9bc7-69a216da89da"). InnerVolumeSpecName "kube-api-access-4zr6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:19:12.032702 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.032626    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44e535-42d7-426e-b7da-b38de1adacf2-plugins-config-volume\") pod \"9f44e535-42d7-426e-b7da-b38de1adacf2\" (UID: \"9f44e535-42d7-426e-b7da-b38de1adacf2\") "
Jul 24 19:19:12.032817 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.032744    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv6wb\" (UniqueName: \"kubernetes.io/projected/9f44e535-42d7-426e-b7da-b38de1adacf2-kube-api-access-nv6wb\") pod \"9f44e535-42d7-426e-b7da-b38de1adacf2\" (UID: \"9f44e535-42d7-426e-b7da-b38de1adacf2\") "
Jul 24 19:19:12.032942 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.032929    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jrvz\" (UniqueName: \"kubernetes.io/projected/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-kube-api-access-2jrvz\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:19:12.032981 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.032945    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:19:12.032981 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.032954    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zr6k\" (UniqueName: \"kubernetes.io/projected/987e6cc1-1149-4a69-9bc7-69a216da89da-kube-api-access-4zr6k\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:19:12.032981 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.032963    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1e614b8-fa6f-4640-8744-ba1d04cea4b2-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:19:12.032981 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.032961    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f44e535-42d7-426e-b7da-b38de1adacf2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f44e535-42d7-426e-b7da-b38de1adacf2" (UID: "9f44e535-42d7-426e-b7da-b38de1adacf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:19:12.033111 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.032972    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987e6cc1-1149-4a69-9bc7-69a216da89da-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:19:12.034629 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.034610    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f44e535-42d7-426e-b7da-b38de1adacf2-kube-api-access-nv6wb" (OuterVolumeSpecName: "kube-api-access-nv6wb") pod "9f44e535-42d7-426e-b7da-b38de1adacf2" (UID: "9f44e535-42d7-426e-b7da-b38de1adacf2"). InnerVolumeSpecName "kube-api-access-nv6wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:19:12.122074 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.122045    2580 generic.go:356] "Generic (PLEG): container finished" podID="987e6cc1-1149-4a69-9bc7-69a216da89da" containerID="800a784c684d6a2f1cccc9b3084b35c599ab83b3a0c192f91439ef484a5f2d20" exitCode=0
Jul 24 19:19:12.122184 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.122089    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"
Jul 24 19:19:12.122184 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.122137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f" event={"ID":"987e6cc1-1149-4a69-9bc7-69a216da89da","Type":"ContainerDied","Data":"800a784c684d6a2f1cccc9b3084b35c599ab83b3a0c192f91439ef484a5f2d20"}
Jul 24 19:19:12.122270 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.122191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f" event={"ID":"987e6cc1-1149-4a69-9bc7-69a216da89da","Type":"ContainerDied","Data":"6728aa0534bc5b7f86cbe372b36096d3d69c91025355ea19ef929282668bdba4"}
Jul 24 19:19:12.122270 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.122225    2580 scope.go:117] "RemoveContainer" containerID="800a784c684d6a2f1cccc9b3084b35c599ab83b3a0c192f91439ef484a5f2d20"
Jul 24 19:19:12.123547 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.123521    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1e614b8-fa6f-4640-8744-ba1d04cea4b2" containerID="5523d7b87dd9eb82c73aa9596ce16b790bbdfdcc3befffcd67a769bdfae4db1e" exitCode=0
Jul 24 19:19:12.123653 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.123555    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"
Jul 24 19:19:12.123653 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.123608    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm" event={"ID":"b1e614b8-fa6f-4640-8744-ba1d04cea4b2","Type":"ContainerDied","Data":"5523d7b87dd9eb82c73aa9596ce16b790bbdfdcc3befffcd67a769bdfae4db1e"}
Jul 24 19:19:12.123770 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.123661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm" event={"ID":"b1e614b8-fa6f-4640-8744-ba1d04cea4b2","Type":"ContainerDied","Data":"919a117ba5e76096c188fbd19cc74be0ff0a6cdb2db6900ac5c0743478818a26"}
Jul 24 19:19:12.125018 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.124995    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f44e535-42d7-426e-b7da-b38de1adacf2" containerID="c2d37e5923200342aeb405066afb222ad56d367423a7856ac421b31cc1a1ac1a" exitCode=0
Jul 24 19:19:12.125138 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.125040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq" event={"ID":"9f44e535-42d7-426e-b7da-b38de1adacf2","Type":"ContainerDied","Data":"c2d37e5923200342aeb405066afb222ad56d367423a7856ac421b31cc1a1ac1a"}
Jul 24 19:19:12.125138 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.125064    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"
Jul 24 19:19:12.125138 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.125078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq" event={"ID":"9f44e535-42d7-426e-b7da-b38de1adacf2","Type":"ContainerDied","Data":"4941b035247e377671a998884c9ce31140338e1e35a1d81dae716d7c2fef6cea"}
Jul 24 19:19:12.133097 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.132961    2580 scope.go:117] "RemoveContainer" containerID="800a784c684d6a2f1cccc9b3084b35c599ab83b3a0c192f91439ef484a5f2d20"
Jul 24 19:19:12.133530 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:19:12.133277    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800a784c684d6a2f1cccc9b3084b35c599ab83b3a0c192f91439ef484a5f2d20\": container with ID starting with 800a784c684d6a2f1cccc9b3084b35c599ab83b3a0c192f91439ef484a5f2d20 not found: ID does not exist" containerID="800a784c684d6a2f1cccc9b3084b35c599ab83b3a0c192f91439ef484a5f2d20"
Jul 24 19:19:12.133530 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.133317    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800a784c684d6a2f1cccc9b3084b35c599ab83b3a0c192f91439ef484a5f2d20"} err="failed to get container status \"800a784c684d6a2f1cccc9b3084b35c599ab83b3a0c192f91439ef484a5f2d20\": rpc error: code = NotFound desc = could not find container \"800a784c684d6a2f1cccc9b3084b35c599ab83b3a0c192f91439ef484a5f2d20\": container with ID starting with 800a784c684d6a2f1cccc9b3084b35c599ab83b3a0c192f91439ef484a5f2d20 not found: ID does not exist"
Jul 24 19:19:12.133530 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.133338    2580 scope.go:117] "RemoveContainer" containerID="5523d7b87dd9eb82c73aa9596ce16b790bbdfdcc3befffcd67a769bdfae4db1e"
Jul 24 19:19:12.133530 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.133472    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv6wb\" (UniqueName: \"kubernetes.io/projected/9f44e535-42d7-426e-b7da-b38de1adacf2-kube-api-access-nv6wb\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:19:12.133530 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.133495    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44e535-42d7-426e-b7da-b38de1adacf2-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:19:12.141345 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.141326    2580 scope.go:117] "RemoveContainer" containerID="5523d7b87dd9eb82c73aa9596ce16b790bbdfdcc3befffcd67a769bdfae4db1e"
Jul 24 19:19:12.141643 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:19:12.141617    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5523d7b87dd9eb82c73aa9596ce16b790bbdfdcc3befffcd67a769bdfae4db1e\": container with ID starting with 5523d7b87dd9eb82c73aa9596ce16b790bbdfdcc3befffcd67a769bdfae4db1e not found: ID does not exist" containerID="5523d7b87dd9eb82c73aa9596ce16b790bbdfdcc3befffcd67a769bdfae4db1e"
Jul 24 19:19:12.141743 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.141645    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5523d7b87dd9eb82c73aa9596ce16b790bbdfdcc3befffcd67a769bdfae4db1e"} err="failed to get container status \"5523d7b87dd9eb82c73aa9596ce16b790bbdfdcc3befffcd67a769bdfae4db1e\": rpc error: code = NotFound desc = could not find container \"5523d7b87dd9eb82c73aa9596ce16b790bbdfdcc3befffcd67a769bdfae4db1e\": container with ID starting with 5523d7b87dd9eb82c73aa9596ce16b790bbdfdcc3befffcd67a769bdfae4db1e not found: ID does not exist"
Jul 24 19:19:12.141743 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.141661    2580 scope.go:117] "RemoveContainer" containerID="c2d37e5923200342aeb405066afb222ad56d367423a7856ac421b31cc1a1ac1a"
Jul 24 19:19:12.149138 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.149122    2580 scope.go:117] "RemoveContainer" containerID="c2d37e5923200342aeb405066afb222ad56d367423a7856ac421b31cc1a1ac1a"
Jul 24 19:19:12.149411 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:19:12.149393    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d37e5923200342aeb405066afb222ad56d367423a7856ac421b31cc1a1ac1a\": container with ID starting with c2d37e5923200342aeb405066afb222ad56d367423a7856ac421b31cc1a1ac1a not found: ID does not exist" containerID="c2d37e5923200342aeb405066afb222ad56d367423a7856ac421b31cc1a1ac1a"
Jul 24 19:19:12.149473 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.149417    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d37e5923200342aeb405066afb222ad56d367423a7856ac421b31cc1a1ac1a"} err="failed to get container status \"c2d37e5923200342aeb405066afb222ad56d367423a7856ac421b31cc1a1ac1a\": rpc error: code = NotFound desc = could not find container \"c2d37e5923200342aeb405066afb222ad56d367423a7856ac421b31cc1a1ac1a\": container with ID starting with c2d37e5923200342aeb405066afb222ad56d367423a7856ac421b31cc1a1ac1a not found: ID does not exist"
Jul 24 19:19:12.150210 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.150193    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"]
Jul 24 19:19:12.156461 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.156441    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5e9947-b67cf485d-b5r8f"]
Jul 24 19:19:12.165718 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.165698    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"]
Jul 24 19:19:12.171925 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.171906    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5e9947-5f679f8944-msjpq"]
Jul 24 19:19:12.188128 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.188106    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"]
Jul 24 19:19:12.191607 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:12.191588    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e9947-855c75bfd6-54rsm"]
Jul 24 19:19:13.197792 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:13.197761    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="987e6cc1-1149-4a69-9bc7-69a216da89da" path="/var/lib/kubelet/pods/987e6cc1-1149-4a69-9bc7-69a216da89da/volumes"
Jul 24 19:19:13.198179 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:13.198101    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f44e535-42d7-426e-b7da-b38de1adacf2" path="/var/lib/kubelet/pods/9f44e535-42d7-426e-b7da-b38de1adacf2/volumes"
Jul 24 19:19:13.198473 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:13.198459    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e614b8-fa6f-4640-8744-ba1d04cea4b2" path="/var/lib/kubelet/pods/b1e614b8-fa6f-4640-8744-ba1d04cea4b2/volumes"
Jul 24 19:19:39.000331 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.000279    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"]
Jul 24 19:19:39.000848 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.000594    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d" podUID="0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9" containerName="maas-api" containerID="cri-o://4d355e6121c3a3990288fa723c2d7ed86161869d9ed8967ca7a55b25c55a46bd" gracePeriod=30
Jul 24 19:19:39.057680 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.057651    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"]
Jul 24 19:19:39.057925 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.057897    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr" podUID="025f68b6-59bb-470b-90eb-541b8c585210" containerName="payload-processing" containerID="cri-o://0e35c23b16a08c04ab765c1f543cf53f4bfa344b71bb0a6282a5cfd4c969e454" gracePeriod=30
Jul 24 19:19:39.067969 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.067940    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"]
Jul 24 19:19:39.068215 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.068170    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2" podUID="9f3ead70-c149-49e2-97b1-b00e5536a456" containerName="payload-pre-processing" containerID="cri-o://f20b6b8af6a30c988401db04c4a3ec5d8f801de681ab4529129e9b9665ee148e" gracePeriod=30
Jul 24 19:19:39.261546 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.261465    2580 generic.go:356] "Generic (PLEG): container finished" podID="0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9" containerID="4d355e6121c3a3990288fa723c2d7ed86161869d9ed8967ca7a55b25c55a46bd" exitCode=0
Jul 24 19:19:39.261664 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.261536    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d" event={"ID":"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9","Type":"ContainerDied","Data":"4d355e6121c3a3990288fa723c2d7ed86161869d9ed8967ca7a55b25c55a46bd"}
Jul 24 19:19:39.263142 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.263118    2580 generic.go:356] "Generic (PLEG): container finished" podID="025f68b6-59bb-470b-90eb-541b8c585210" containerID="0e35c23b16a08c04ab765c1f543cf53f4bfa344b71bb0a6282a5cfd4c969e454" exitCode=0
Jul 24 19:19:39.263246 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.263226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr" event={"ID":"025f68b6-59bb-470b-90eb-541b8c585210","Type":"ContainerDied","Data":"0e35c23b16a08c04ab765c1f543cf53f4bfa344b71bb0a6282a5cfd4c969e454"}
Jul 24 19:19:39.264852 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.264785    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f3ead70-c149-49e2-97b1-b00e5536a456" containerID="f20b6b8af6a30c988401db04c4a3ec5d8f801de681ab4529129e9b9665ee148e" exitCode=0
Jul 24 19:19:39.264953 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.264891    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2" event={"ID":"9f3ead70-c149-49e2-97b1-b00e5536a456","Type":"ContainerDied","Data":"f20b6b8af6a30c988401db04c4a3ec5d8f801de681ab4529129e9b9665ee148e"}
Jul 24 19:19:39.336801 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.336776    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"
Jul 24 19:19:39.361833 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.361804    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"
Jul 24 19:19:39.370729 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.370703    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:19:39.376326 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.376212    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-maas-api-tls\") pod \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\" (UID: \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\") "
Jul 24 19:19:39.376326 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.376271    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025f68b6-59bb-470b-90eb-541b8c585210-plugins-config-volume\") pod \"025f68b6-59bb-470b-90eb-541b8c585210\" (UID: \"025f68b6-59bb-470b-90eb-541b8c585210\") "
Jul 24 19:19:39.376582 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.376353    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-openshift-service-ca\") pod \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\" (UID: \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\") "
Jul 24 19:19:39.376582 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.376425    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6p2k\" (UniqueName: \"kubernetes.io/projected/9f3ead70-c149-49e2-97b1-b00e5536a456-kube-api-access-v6p2k\") pod \"9f3ead70-c149-49e2-97b1-b00e5536a456\" (UID: \"9f3ead70-c149-49e2-97b1-b00e5536a456\") "
Jul 24 19:19:39.376582 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.376463    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mthb\" (UniqueName: \"kubernetes.io/projected/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-kube-api-access-5mthb\") pod \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\" (UID: \"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9\") "
Jul 24 19:19:39.376582 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.376486    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjdp6\" (UniqueName: \"kubernetes.io/projected/025f68b6-59bb-470b-90eb-541b8c585210-kube-api-access-gjdp6\") pod \"025f68b6-59bb-470b-90eb-541b8c585210\" (UID: \"025f68b6-59bb-470b-90eb-541b8c585210\") "
Jul 24 19:19:39.376582 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.376524    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3ead70-c149-49e2-97b1-b00e5536a456-plugins-config-volume\") pod \"9f3ead70-c149-49e2-97b1-b00e5536a456\" (UID: \"9f3ead70-c149-49e2-97b1-b00e5536a456\") "
Jul 24 19:19:39.377188 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.377159    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f3ead70-c149-49e2-97b1-b00e5536a456-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f3ead70-c149-49e2-97b1-b00e5536a456" (UID: "9f3ead70-c149-49e2-97b1-b00e5536a456"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:19:39.377951 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.377488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9" (UID: "0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:19:39.377951 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.377918    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025f68b6-59bb-470b-90eb-541b8c585210-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "025f68b6-59bb-470b-90eb-541b8c585210" (UID: "025f68b6-59bb-470b-90eb-541b8c585210"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:19:39.379642 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.379524    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9" (UID: "0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:19:39.380803 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.380773    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f3ead70-c149-49e2-97b1-b00e5536a456-kube-api-access-v6p2k" (OuterVolumeSpecName: "kube-api-access-v6p2k") pod "9f3ead70-c149-49e2-97b1-b00e5536a456" (UID: "9f3ead70-c149-49e2-97b1-b00e5536a456"). InnerVolumeSpecName "kube-api-access-v6p2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:19:39.380977 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.380956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-kube-api-access-5mthb" (OuterVolumeSpecName: "kube-api-access-5mthb") pod "0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9" (UID: "0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9"). InnerVolumeSpecName "kube-api-access-5mthb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:19:39.381241 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.381211    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025f68b6-59bb-470b-90eb-541b8c585210-kube-api-access-gjdp6" (OuterVolumeSpecName: "kube-api-access-gjdp6") pod "025f68b6-59bb-470b-90eb-541b8c585210" (UID: "025f68b6-59bb-470b-90eb-541b8c585210"). InnerVolumeSpecName "kube-api-access-gjdp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:19:39.477500 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.477469    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:19:39.477500 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.477497    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6p2k\" (UniqueName: \"kubernetes.io/projected/9f3ead70-c149-49e2-97b1-b00e5536a456-kube-api-access-v6p2k\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:19:39.477500 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.477506    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mthb\" (UniqueName: \"kubernetes.io/projected/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-kube-api-access-5mthb\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:19:39.477726 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.477515    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjdp6\" (UniqueName: \"kubernetes.io/projected/025f68b6-59bb-470b-90eb-541b8c585210-kube-api-access-gjdp6\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:19:39.477726 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.477527    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3ead70-c149-49e2-97b1-b00e5536a456-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:19:39.477726 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.477541    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:19:39.477726 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:39.477549    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025f68b6-59bb-470b-90eb-541b8c585210-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:19:40.271667 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.271582    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"
Jul 24 19:19:40.272130 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.271585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d" event={"ID":"0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9","Type":"ContainerDied","Data":"bf86765e27acbcc5673bee94d973347ce3288602187f7df890acee18b8090583"}
Jul 24 19:19:40.272130 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.271715    2580 scope.go:117] "RemoveContainer" containerID="4d355e6121c3a3990288fa723c2d7ed86161869d9ed8967ca7a55b25c55a46bd"
Jul 24 19:19:40.273102 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.272874    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"
Jul 24 19:19:40.273102 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.272887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr" event={"ID":"025f68b6-59bb-470b-90eb-541b8c585210","Type":"ContainerDied","Data":"05771d32d9c1aab9f4c1af5e7b564dd74728a27c8188f058a09b4e055ce61cf9"}
Jul 24 19:19:40.274230 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.274206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2" event={"ID":"9f3ead70-c149-49e2-97b1-b00e5536a456","Type":"ContainerDied","Data":"772b761d7d240a366a412e12766000acdd032f61f42ea9154d415d8e7c9d057a"}
Jul 24 19:19:40.274350 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.274237    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"
Jul 24 19:19:40.282269 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.282254    2580 scope.go:117] "RemoveContainer" containerID="0e35c23b16a08c04ab765c1f543cf53f4bfa344b71bb0a6282a5cfd4c969e454"
Jul 24 19:19:40.290429 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.290409    2580 scope.go:117] "RemoveContainer" containerID="f20b6b8af6a30c988401db04c4a3ec5d8f801de681ab4529129e9b9665ee148e"
Jul 24 19:19:40.296422 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.296400    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"]
Jul 24 19:19:40.299823 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.299803    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a90216-c689f9875-wtf9d"]
Jul 24 19:19:40.309485 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.309463    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"]
Jul 24 19:19:40.311693 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.311674    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a90216-6c8d885fb-45jb2"]
Jul 24 19:19:40.321722 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.321693    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"]
Jul 24 19:19:40.323363 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:40.323344    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a90216-7d687685d8-4qmpr"]
Jul 24 19:19:41.198103 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:41.198068    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="025f68b6-59bb-470b-90eb-541b8c585210" path="/var/lib/kubelet/pods/025f68b6-59bb-470b-90eb-541b8c585210/volumes"
Jul 24 19:19:41.198502 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:41.198488    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9" path="/var/lib/kubelet/pods/0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9/volumes"
Jul 24 19:19:41.198808 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:41.198797    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f3ead70-c149-49e2-97b1-b00e5536a456" path="/var/lib/kubelet/pods/9f3ead70-c149-49e2-97b1-b00e5536a456/volumes"
Jul 24 19:19:54.414339 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.414294    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"]
Jul 24 19:19:54.414916 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.414894    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987e6cc1-1149-4a69-9bc7-69a216da89da" containerName="payload-processing"
Jul 24 19:19:54.414961 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.414921    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="987e6cc1-1149-4a69-9bc7-69a216da89da" containerName="payload-processing"
Jul 24 19:19:54.414961 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.414935    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e614b8-fa6f-4640-8744-ba1d04cea4b2" containerName="maas-api"
Jul 24 19:19:54.414961 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.414945    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e614b8-fa6f-4640-8744-ba1d04cea4b2" containerName="maas-api"
Jul 24 19:19:54.415056 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.414973    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9" containerName="maas-api"
Jul 24 19:19:54.415056 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.414981    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9" containerName="maas-api"
Jul 24 19:19:54.415056 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.414992    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f3ead70-c149-49e2-97b1-b00e5536a456" containerName="payload-pre-processing"
Jul 24 19:19:54.415056 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.415001    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f3ead70-c149-49e2-97b1-b00e5536a456" containerName="payload-pre-processing"
Jul 24 19:19:54.415056 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.415019    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f44e535-42d7-426e-b7da-b38de1adacf2" containerName="payload-pre-processing"
Jul 24 19:19:54.415056 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.415028    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f44e535-42d7-426e-b7da-b38de1adacf2" containerName="payload-pre-processing"
Jul 24 19:19:54.415056 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.415037    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025f68b6-59bb-470b-90eb-541b8c585210" containerName="payload-processing"
Jul 24 19:19:54.415056 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.415045    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="025f68b6-59bb-470b-90eb-541b8c585210" containerName="payload-processing"
Jul 24 19:19:54.415292 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.415138    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e614b8-fa6f-4640-8744-ba1d04cea4b2" containerName="maas-api"
Jul 24 19:19:54.415292 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.415152    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="025f68b6-59bb-470b-90eb-541b8c585210" containerName="payload-processing"
Jul 24 19:19:54.415292 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.415165    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f3ead70-c149-49e2-97b1-b00e5536a456" containerName="payload-pre-processing"
Jul 24 19:19:54.415292 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.415175    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f44e535-42d7-426e-b7da-b38de1adacf2" containerName="payload-pre-processing"
Jul 24 19:19:54.415292 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.415185    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="987e6cc1-1149-4a69-9bc7-69a216da89da" containerName="payload-processing"
Jul 24 19:19:54.415292 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.415195    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cb2bad8-8c3d-4705-a6d3-6a674bc6acf9" containerName="maas-api"
Jul 24 19:19:54.418080 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.418063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:19:54.420072 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.420052    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-65184f-serving-cert\""
Jul 24 19:19:54.439485 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.439461    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"]
Jul 24 19:19:54.500507 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.500479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee47147c-3f74-46d1-9b47-0a7062cc0db6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-65184f-56d55d9666-njtlm\" (UID: \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:19:54.500620 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.500525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee47147c-3f74-46d1-9b47-0a7062cc0db6-maas-api-tls\") pod \"maas-api-e2e-shared-a-65184f-56d55d9666-njtlm\" (UID: \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:19:54.500620 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.500546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv6pr\" (UniqueName: \"kubernetes.io/projected/ee47147c-3f74-46d1-9b47-0a7062cc0db6-kube-api-access-sv6pr\") pod \"maas-api-e2e-shared-a-65184f-56d55d9666-njtlm\" (UID: \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:19:54.600893 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.600863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee47147c-3f74-46d1-9b47-0a7062cc0db6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-65184f-56d55d9666-njtlm\" (UID: \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:19:54.601005 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.600909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee47147c-3f74-46d1-9b47-0a7062cc0db6-maas-api-tls\") pod \"maas-api-e2e-shared-a-65184f-56d55d9666-njtlm\" (UID: \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:19:54.601005 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.600926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv6pr\" (UniqueName: \"kubernetes.io/projected/ee47147c-3f74-46d1-9b47-0a7062cc0db6-kube-api-access-sv6pr\") pod \"maas-api-e2e-shared-a-65184f-56d55d9666-njtlm\" (UID: \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:19:54.601625 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.601600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee47147c-3f74-46d1-9b47-0a7062cc0db6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-65184f-56d55d9666-njtlm\" (UID: \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:19:54.603297 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.603276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee47147c-3f74-46d1-9b47-0a7062cc0db6-maas-api-tls\") pod \"maas-api-e2e-shared-a-65184f-56d55d9666-njtlm\" (UID: \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:19:54.609073 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.609050    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv6pr\" (UniqueName: \"kubernetes.io/projected/ee47147c-3f74-46d1-9b47-0a7062cc0db6-kube-api-access-sv6pr\") pod \"maas-api-e2e-shared-a-65184f-56d55d9666-njtlm\" (UID: \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:19:54.727852 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.727777    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:19:54.856825 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:54.856798    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"]
Jul 24 19:19:54.858707 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:19:54.858679    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee47147c_3f74_46d1_9b47_0a7062cc0db6.slice/crio-ccc8b78fe9f6a5bd6b09c6602fd3001e740308204886004652be944aa4f1397d WatchSource:0}: Error finding container ccc8b78fe9f6a5bd6b09c6602fd3001e740308204886004652be944aa4f1397d: Status 404 returned error can't find the container with id ccc8b78fe9f6a5bd6b09c6602fd3001e740308204886004652be944aa4f1397d
Jul 24 19:19:55.201494 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.201458    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"]
Jul 24 19:19:55.206314 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.206297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"
Jul 24 19:19:55.208531 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.208509    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-65184f-dockercfg-vpvtq\""
Jul 24 19:19:55.208638 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.208548    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-65184f\""
Jul 24 19:19:55.212607 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.212582    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"]
Jul 24 19:19:55.307110 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.307084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ffc07c-c370-461f-9d6f-39d1e4b563f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm\" (UID: \"04ffc07c-c370-461f-9d6f-39d1e4b563f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"
Jul 24 19:19:55.307271 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.307241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x72zv\" (UniqueName: \"kubernetes.io/projected/04ffc07c-c370-461f-9d6f-39d1e4b563f8-kube-api-access-x72zv\") pod \"payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm\" (UID: \"04ffc07c-c370-461f-9d6f-39d1e4b563f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"
Jul 24 19:19:55.307820 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.307801    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"]
Jul 24 19:19:55.311415 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.311397    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"
Jul 24 19:19:55.321760 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.321661    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"]
Jul 24 19:19:55.349926 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.349899    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm" event={"ID":"ee47147c-3f74-46d1-9b47-0a7062cc0db6","Type":"ContainerStarted","Data":"ccc8b78fe9f6a5bd6b09c6602fd3001e740308204886004652be944aa4f1397d"}
Jul 24 19:19:55.408446 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.408393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbm4f\" (UniqueName: \"kubernetes.io/projected/0690cb74-7813-4c14-895c-f1d6cde543db-kube-api-access-nbm4f\") pod \"payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8\" (UID: \"0690cb74-7813-4c14-895c-f1d6cde543db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"
Jul 24 19:19:55.408613 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.408500    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0690cb74-7813-4c14-895c-f1d6cde543db-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8\" (UID: \"0690cb74-7813-4c14-895c-f1d6cde543db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"
Jul 24 19:19:55.408613 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.408527    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x72zv\" (UniqueName: \"kubernetes.io/projected/04ffc07c-c370-461f-9d6f-39d1e4b563f8-kube-api-access-x72zv\") pod \"payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm\" (UID: \"04ffc07c-c370-461f-9d6f-39d1e4b563f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"
Jul 24 19:19:55.408613 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.408600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ffc07c-c370-461f-9d6f-39d1e4b563f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm\" (UID: \"04ffc07c-c370-461f-9d6f-39d1e4b563f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"
Jul 24 19:19:55.409310 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.409284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ffc07c-c370-461f-9d6f-39d1e4b563f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm\" (UID: \"04ffc07c-c370-461f-9d6f-39d1e4b563f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"
Jul 24 19:19:55.415675 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.415657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x72zv\" (UniqueName: \"kubernetes.io/projected/04ffc07c-c370-461f-9d6f-39d1e4b563f8-kube-api-access-x72zv\") pod \"payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm\" (UID: \"04ffc07c-c370-461f-9d6f-39d1e4b563f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"
Jul 24 19:19:55.509137 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.509057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0690cb74-7813-4c14-895c-f1d6cde543db-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8\" (UID: \"0690cb74-7813-4c14-895c-f1d6cde543db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"
Jul 24 19:19:55.509292 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.509159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbm4f\" (UniqueName: \"kubernetes.io/projected/0690cb74-7813-4c14-895c-f1d6cde543db-kube-api-access-nbm4f\") pod \"payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8\" (UID: \"0690cb74-7813-4c14-895c-f1d6cde543db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"
Jul 24 19:19:55.509812 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.509791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0690cb74-7813-4c14-895c-f1d6cde543db-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8\" (UID: \"0690cb74-7813-4c14-895c-f1d6cde543db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"
Jul 24 19:19:55.516259 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.516236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbm4f\" (UniqueName: \"kubernetes.io/projected/0690cb74-7813-4c14-895c-f1d6cde543db-kube-api-access-nbm4f\") pod \"payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8\" (UID: \"0690cb74-7813-4c14-895c-f1d6cde543db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"
Jul 24 19:19:55.518039 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.518021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"
Jul 24 19:19:55.624840 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.624811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"
Jul 24 19:19:55.656810 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.656773    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"]
Jul 24 19:19:55.661636 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:19:55.661600    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ffc07c_c370_461f_9d6f_39d1e4b563f8.slice/crio-1345f3b3dc03f95192e040585ccd7b4393caa2ad60a0a7bf00a3f4451657097f WatchSource:0}: Error finding container 1345f3b3dc03f95192e040585ccd7b4393caa2ad60a0a7bf00a3f4451657097f: Status 404 returned error can't find the container with id 1345f3b3dc03f95192e040585ccd7b4393caa2ad60a0a7bf00a3f4451657097f
Jul 24 19:19:55.772894 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:55.772739    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"]
Jul 24 19:19:56.356595 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:56.356556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm" event={"ID":"ee47147c-3f74-46d1-9b47-0a7062cc0db6","Type":"ContainerStarted","Data":"40b07e1fe045603f80e951f29534401b113a82428389255ac92e789fa6811a9f"}
Jul 24 19:19:56.356806 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:56.356625    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:19:56.358236 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:56.358207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm" event={"ID":"04ffc07c-c370-461f-9d6f-39d1e4b563f8","Type":"ContainerStarted","Data":"3f9e4ebcdaa0c3e1c073791d6cd5a6d6ead9a60af3c0a65b55e85b5661e4ff6a"}
Jul 24 19:19:56.358373 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:56.358256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm" event={"ID":"04ffc07c-c370-461f-9d6f-39d1e4b563f8","Type":"ContainerStarted","Data":"1345f3b3dc03f95192e040585ccd7b4393caa2ad60a0a7bf00a3f4451657097f"}
Jul 24 19:19:56.358373 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:56.358323    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"
Jul 24 19:19:56.359756 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:56.359732    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8" event={"ID":"0690cb74-7813-4c14-895c-f1d6cde543db","Type":"ContainerStarted","Data":"c96071a68194b97733cbe8ae76fe02a41dba238b3569f737147db437220529f6"}
Jul 24 19:19:56.359868 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:56.359768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8" event={"ID":"0690cb74-7813-4c14-895c-f1d6cde543db","Type":"ContainerStarted","Data":"37ee1e9a711d764657abe31f83e021891997c5af9775105bbe69f452669b5904"}
Jul 24 19:19:56.359868 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:56.359858    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"
Jul 24 19:19:56.373704 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:56.373663    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm" podStartSLOduration=1.9313843130000001 podStartE2EDuration="2.373647769s" totalImagesPullingTime="442.263456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:19:56.371705432 +0000 UTC m=+2395.747364725" watchObservedRunningTime="2026-07-24 19:19:56.373647769 +0000 UTC m=+2395.749307066"
Jul 24 19:19:56.387638 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:56.387595    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8" podStartSLOduration=1.387581608 podStartE2EDuration="1.387581608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:19:56.385141651 +0000 UTC m=+2395.760800947" watchObservedRunningTime="2026-07-24 19:19:56.387581608 +0000 UTC m=+2395.763240882"
Jul 24 19:19:56.399751 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:19:56.399710    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm" podStartSLOduration=1.3996966180000001 podStartE2EDuration="1.399696618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:19:56.397932747 +0000 UTC m=+2395.773592044" watchObservedRunningTime="2026-07-24 19:19:56.399696618 +0000 UTC m=+2395.775355915"
Jul 24 19:20:01.267981 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:01.267952    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:20:01.270786 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:01.270765    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:20:02.347214 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:02.347185    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"]
Jul 24 19:20:02.352740 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:02.352720    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:02.355024 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:02.355003    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-d391c5-serving-cert\""
Jul 24 19:20:02.363415 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:02.363364    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"]
Jul 24 19:20:02.370646 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:02.370627    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:20:02.469476 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:02.469441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/962db09a-a11e-4b9a-ae6f-29981e81150c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8\" (UID: \"962db09a-a11e-4b9a-ae6f-29981e81150c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:02.469658 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:02.469547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/962db09a-a11e-4b9a-ae6f-29981e81150c-maas-api-tls\") pod \"maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8\" (UID: \"962db09a-a11e-4b9a-ae6f-29981e81150c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:02.469658 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:02.469579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn6hz\" (UniqueName: \"kubernetes.io/projected/962db09a-a11e-4b9a-ae6f-29981e81150c-kube-api-access-vn6hz\") pod \"maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8\" (UID: \"962db09a-a11e-4b9a-ae6f-29981e81150c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:02.569982 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:02.569951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/962db09a-a11e-4b9a-ae6f-29981e81150c-maas-api-tls\") pod \"maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8\" (UID: \"962db09a-a11e-4b9a-ae6f-29981e81150c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:02.570141 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:02.569984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn6hz\" (UniqueName: \"kubernetes.io/projected/962db09a-a11e-4b9a-ae6f-29981e81150c-kube-api-access-vn6hz\") pod \"maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8\" (UID: \"962db09a-a11e-4b9a-ae6f-29981e81150c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:02.570141 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:02.570043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/962db09a-a11e-4b9a-ae6f-29981e81150c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8\" (UID: \"962db09a-a11e-4b9a-ae6f-29981e81150c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:02.570141 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:20:02.570078    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-serving-cert: secret "maas-api-e2e-shared-b-d391c5-serving-cert" not found
Jul 24 19:20:02.570340 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:20:02.570152    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/962db09a-a11e-4b9a-ae6f-29981e81150c-maas-api-tls podName:962db09a-a11e-4b9a-ae6f-29981e81150c nodeName:}" failed. No retries permitted until 2026-07-24 19:20:03.070134506 +0000 UTC m=+2402.445793794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/962db09a-a11e-4b9a-ae6f-29981e81150c-maas-api-tls") pod "maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8" (UID: "962db09a-a11e-4b9a-ae6f-29981e81150c") : secret "maas-api-e2e-shared-b-d391c5-serving-cert" not found
Jul 24 19:20:02.570660 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:02.570641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/962db09a-a11e-4b9a-ae6f-29981e81150c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8\" (UID: \"962db09a-a11e-4b9a-ae6f-29981e81150c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:02.578115 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:02.578094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn6hz\" (UniqueName: \"kubernetes.io/projected/962db09a-a11e-4b9a-ae6f-29981e81150c-kube-api-access-vn6hz\") pod \"maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8\" (UID: \"962db09a-a11e-4b9a-ae6f-29981e81150c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:03.074358 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.074313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/962db09a-a11e-4b9a-ae6f-29981e81150c-maas-api-tls\") pod \"maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8\" (UID: \"962db09a-a11e-4b9a-ae6f-29981e81150c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:03.074536 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:20:03.074492    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-serving-cert: secret "maas-api-e2e-shared-b-d391c5-serving-cert" not found
Jul 24 19:20:03.074609 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:20:03.074554    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/962db09a-a11e-4b9a-ae6f-29981e81150c-maas-api-tls podName:962db09a-a11e-4b9a-ae6f-29981e81150c nodeName:}" failed. No retries permitted until 2026-07-24 19:20:04.074539702 +0000 UTC m=+2403.450198976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/962db09a-a11e-4b9a-ae6f-29981e81150c-maas-api-tls") pod "maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8" (UID: "962db09a-a11e-4b9a-ae6f-29981e81150c") : secret "maas-api-e2e-shared-b-d391c5-serving-cert" not found
Jul 24 19:20:03.092343 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.092319    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"]
Jul 24 19:20:03.095873 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.095850    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"
Jul 24 19:20:03.100421 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.100375    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-d391c5-dockercfg-pw5xz\""
Jul 24 19:20:03.100421 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.100414    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-d391c5\""
Jul 24 19:20:03.113893 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.113871    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"]
Jul 24 19:20:03.122985 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.122960    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"]
Jul 24 19:20:03.126334 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.126314    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"
Jul 24 19:20:03.138715 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.138696    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"]
Jul 24 19:20:03.175793 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.175764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jckgs\" (UniqueName: \"kubernetes.io/projected/35710028-b82f-4f57-a927-abcd5d2bd0d5-kube-api-access-jckgs\") pod \"payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx\" (UID: \"35710028-b82f-4f57-a927-abcd5d2bd0d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"
Jul 24 19:20:03.175901 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.175812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35710028-b82f-4f57-a927-abcd5d2bd0d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx\" (UID: \"35710028-b82f-4f57-a927-abcd5d2bd0d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"
Jul 24 19:20:03.175901 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.175879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z22lx\" (UniqueName: \"kubernetes.io/projected/15af558e-bfc3-459c-9624-4f5cc872b03f-kube-api-access-z22lx\") pod \"payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn\" (UID: \"15af558e-bfc3-459c-9624-4f5cc872b03f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"
Jul 24 19:20:03.176003 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.175985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15af558e-bfc3-459c-9624-4f5cc872b03f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn\" (UID: \"15af558e-bfc3-459c-9624-4f5cc872b03f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"
Jul 24 19:20:03.277193 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.277163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jckgs\" (UniqueName: \"kubernetes.io/projected/35710028-b82f-4f57-a927-abcd5d2bd0d5-kube-api-access-jckgs\") pod \"payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx\" (UID: \"35710028-b82f-4f57-a927-abcd5d2bd0d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"
Jul 24 19:20:03.277339 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.277215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35710028-b82f-4f57-a927-abcd5d2bd0d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx\" (UID: \"35710028-b82f-4f57-a927-abcd5d2bd0d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"
Jul 24 19:20:03.277339 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.277261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z22lx\" (UniqueName: \"kubernetes.io/projected/15af558e-bfc3-459c-9624-4f5cc872b03f-kube-api-access-z22lx\") pod \"payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn\" (UID: \"15af558e-bfc3-459c-9624-4f5cc872b03f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"
Jul 24 19:20:03.277452 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.277343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15af558e-bfc3-459c-9624-4f5cc872b03f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn\" (UID: \"15af558e-bfc3-459c-9624-4f5cc872b03f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"
Jul 24 19:20:03.278218 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.278187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15af558e-bfc3-459c-9624-4f5cc872b03f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn\" (UID: \"15af558e-bfc3-459c-9624-4f5cc872b03f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"
Jul 24 19:20:03.278361 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.278187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35710028-b82f-4f57-a927-abcd5d2bd0d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx\" (UID: \"35710028-b82f-4f57-a927-abcd5d2bd0d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"
Jul 24 19:20:03.286158 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.286134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jckgs\" (UniqueName: \"kubernetes.io/projected/35710028-b82f-4f57-a927-abcd5d2bd0d5-kube-api-access-jckgs\") pod \"payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx\" (UID: \"35710028-b82f-4f57-a927-abcd5d2bd0d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"
Jul 24 19:20:03.286269 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.286178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z22lx\" (UniqueName: \"kubernetes.io/projected/15af558e-bfc3-459c-9624-4f5cc872b03f-kube-api-access-z22lx\") pod \"payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn\" (UID: \"15af558e-bfc3-459c-9624-4f5cc872b03f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"
Jul 24 19:20:03.406698 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.406619    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"
Jul 24 19:20:03.437979 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.437953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"
Jul 24 19:20:03.564090 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.564056    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"]
Jul 24 19:20:03.566469 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:20:03.566440    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35710028_b82f_4f57_a927_abcd5d2bd0d5.slice/crio-ed5551cfc06d3674fbafff13587568bf4ec56857b27a5e59334cf482b61a60dc WatchSource:0}: Error finding container ed5551cfc06d3674fbafff13587568bf4ec56857b27a5e59334cf482b61a60dc: Status 404 returned error can't find the container with id ed5551cfc06d3674fbafff13587568bf4ec56857b27a5e59334cf482b61a60dc
Jul 24 19:20:03.591897 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:03.591864    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"]
Jul 24 19:20:03.595020 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:20:03.594685    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15af558e_bfc3_459c_9624_4f5cc872b03f.slice/crio-c735737413446ec1ac54ce9a917afeb5e0c7b7d911e82419b9ddbff23a6e02cd WatchSource:0}: Error finding container c735737413446ec1ac54ce9a917afeb5e0c7b7d911e82419b9ddbff23a6e02cd: Status 404 returned error can't find the container with id c735737413446ec1ac54ce9a917afeb5e0c7b7d911e82419b9ddbff23a6e02cd
Jul 24 19:20:04.084117 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:04.084076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/962db09a-a11e-4b9a-ae6f-29981e81150c-maas-api-tls\") pod \"maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8\" (UID: \"962db09a-a11e-4b9a-ae6f-29981e81150c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:04.087016 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:04.086977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/962db09a-a11e-4b9a-ae6f-29981e81150c-maas-api-tls\") pod \"maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8\" (UID: \"962db09a-a11e-4b9a-ae6f-29981e81150c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:04.163898 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:04.163858    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:04.306975 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:04.306948    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"]
Jul 24 19:20:04.308907 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:20:04.308884    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962db09a_a11e_4b9a_ae6f_29981e81150c.slice/crio-6831da63e7aebf0468a6ab3d836f56a382239cb301db16323f7bb7e1b176ad63 WatchSource:0}: Error finding container 6831da63e7aebf0468a6ab3d836f56a382239cb301db16323f7bb7e1b176ad63: Status 404 returned error can't find the container with id 6831da63e7aebf0468a6ab3d836f56a382239cb301db16323f7bb7e1b176ad63
Jul 24 19:20:04.400624 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:04.400543    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8" event={"ID":"962db09a-a11e-4b9a-ae6f-29981e81150c","Type":"ContainerStarted","Data":"6831da63e7aebf0468a6ab3d836f56a382239cb301db16323f7bb7e1b176ad63"}
Jul 24 19:20:04.402151 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:04.402130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn" event={"ID":"15af558e-bfc3-459c-9624-4f5cc872b03f","Type":"ContainerStarted","Data":"ff48e2d812d8e1077f04850e938670bcf40ac19a3e58c0271e78d64b658a0e33"}
Jul 24 19:20:04.402219 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:04.402170    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn" event={"ID":"15af558e-bfc3-459c-9624-4f5cc872b03f","Type":"ContainerStarted","Data":"c735737413446ec1ac54ce9a917afeb5e0c7b7d911e82419b9ddbff23a6e02cd"}
Jul 24 19:20:04.402280 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:04.402226    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"
Jul 24 19:20:04.403707 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:04.403678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx" event={"ID":"35710028-b82f-4f57-a927-abcd5d2bd0d5","Type":"ContainerStarted","Data":"f75fc127e839636950a5b044ab370b25700a78c3f1199fbb73b6e06f1eb81137"}
Jul 24 19:20:04.403814 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:04.403724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx" event={"ID":"35710028-b82f-4f57-a927-abcd5d2bd0d5","Type":"ContainerStarted","Data":"ed5551cfc06d3674fbafff13587568bf4ec56857b27a5e59334cf482b61a60dc"}
Jul 24 19:20:04.403814 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:04.403760    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"
Jul 24 19:20:04.417104 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:04.416791    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn" podStartSLOduration=1.4167736450000001 podStartE2EDuration="1.416773645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:20:04.416265427 +0000 UTC m=+2403.791924732" watchObservedRunningTime="2026-07-24 19:20:04.416773645 +0000 UTC m=+2403.792432944"
Jul 24 19:20:04.433346 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:04.433297    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx" podStartSLOduration=1.433278977 podStartE2EDuration="1.433278977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:20:04.43022126 +0000 UTC m=+2403.805880557" watchObservedRunningTime="2026-07-24 19:20:04.433278977 +0000 UTC m=+2403.808938276"
Jul 24 19:20:05.410394 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:05.410342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8" event={"ID":"962db09a-a11e-4b9a-ae6f-29981e81150c","Type":"ContainerStarted","Data":"c60881a6f72febd8cba506d615bf5c725293146aaf1bb1aae03a78f7d94e31b5"}
Jul 24 19:20:05.410585 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:05.410433    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:05.425168 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:05.425094    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8" podStartSLOduration=3.055870836 podStartE2EDuration="3.425077153s" totalImagesPullingTime="369.206317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:20:05.424665933 +0000 UTC m=+2404.800325251" watchObservedRunningTime="2026-07-24 19:20:05.425077153 +0000 UTC m=+2404.800736451"
Jul 24 19:20:07.367172 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:07.367139    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"
Jul 24 19:20:07.367584 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:07.367215    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"
Jul 24 19:20:11.422902 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:11.422873    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:20:15.412413 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:15.412361    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"
Jul 24 19:20:15.412779 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:20:15.412560    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"
Jul 24 19:25:01.298184 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:01.298118    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:25:01.301572 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:01.301549    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:25:46.378027 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:46.377992    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7690-139959310141456"]
Jul 24 19:25:46.381554 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:46.381537    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7690-139959310141456"
Jul 24 19:25:46.383646 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:46.383626    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9k27v\""
Jul 24 19:25:46.387836 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:46.387814    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7690-139959310141456"]
Jul 24 19:25:46.440893 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:46.440868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkdch\" (UniqueName: \"kubernetes.io/projected/e20af6e0-f6f0-42df-8bba-67681dbc4724-kube-api-access-jkdch\") pod \"test-curl-7690-139959310141456\" (UID: \"e20af6e0-f6f0-42df-8bba-67681dbc4724\") " pod="opendatahub/test-curl-7690-139959310141456"
Jul 24 19:25:46.541844 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:46.541816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkdch\" (UniqueName: \"kubernetes.io/projected/e20af6e0-f6f0-42df-8bba-67681dbc4724-kube-api-access-jkdch\") pod \"test-curl-7690-139959310141456\" (UID: \"e20af6e0-f6f0-42df-8bba-67681dbc4724\") " pod="opendatahub/test-curl-7690-139959310141456"
Jul 24 19:25:46.550689 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:46.550661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkdch\" (UniqueName: \"kubernetes.io/projected/e20af6e0-f6f0-42df-8bba-67681dbc4724-kube-api-access-jkdch\") pod \"test-curl-7690-139959310141456\" (UID: \"e20af6e0-f6f0-42df-8bba-67681dbc4724\") " pod="opendatahub/test-curl-7690-139959310141456"
Jul 24 19:25:46.692355 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:46.692279    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7690-139959310141456"
Jul 24 19:25:46.821568 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:46.821542    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7690-139959310141456"]
Jul 24 19:25:46.824436 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:25:46.824411    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode20af6e0_f6f0_42df_8bba_67681dbc4724.slice/crio-8f38cac2ddf02ee3131fe8f9c8b795dd080ee3d7c1960820cbd90e69817f196b WatchSource:0}: Error finding container 8f38cac2ddf02ee3131fe8f9c8b795dd080ee3d7c1960820cbd90e69817f196b: Status 404 returned error can't find the container with id 8f38cac2ddf02ee3131fe8f9c8b795dd080ee3d7c1960820cbd90e69817f196b
Jul 24 19:25:46.826099 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:46.826080    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 19:25:47.062763 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:47.062684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7690-139959310141456" event={"ID":"e20af6e0-f6f0-42df-8bba-67681dbc4724","Type":"ContainerStarted","Data":"8f38cac2ddf02ee3131fe8f9c8b795dd080ee3d7c1960820cbd90e69817f196b"}
Jul 24 19:25:48.069470 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:48.069433    2580 generic.go:356] "Generic (PLEG): container finished" podID="e20af6e0-f6f0-42df-8bba-67681dbc4724" containerID="c6d5f9fc59d56973404eee9d6054327eb85e64fe30750c07af49f2ff2535d20f" exitCode=0
Jul 24 19:25:48.069830 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:48.069500    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7690-139959310141456" event={"ID":"e20af6e0-f6f0-42df-8bba-67681dbc4724","Type":"ContainerDied","Data":"c6d5f9fc59d56973404eee9d6054327eb85e64fe30750c07af49f2ff2535d20f"}
Jul 24 19:25:49.209189 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:49.209159    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7690-139959310141456"
Jul 24 19:25:49.242257 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:49.242236    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7690-139959310141456_e20af6e0-f6f0-42df-8bba-67681dbc4724/test-curl-7690-139959310141456/0.log"
Jul 24 19:25:49.264770 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:49.264647    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkdch\" (UniqueName: \"kubernetes.io/projected/e20af6e0-f6f0-42df-8bba-67681dbc4724-kube-api-access-jkdch\") pod \"e20af6e0-f6f0-42df-8bba-67681dbc4724\" (UID: \"e20af6e0-f6f0-42df-8bba-67681dbc4724\") "
Jul 24 19:25:49.266586 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:49.266564    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7690-139959310141456"]
Jul 24 19:25:49.266807 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:49.266787    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20af6e0-f6f0-42df-8bba-67681dbc4724-kube-api-access-jkdch" (OuterVolumeSpecName: "kube-api-access-jkdch") pod "e20af6e0-f6f0-42df-8bba-67681dbc4724" (UID: "e20af6e0-f6f0-42df-8bba-67681dbc4724"). InnerVolumeSpecName "kube-api-access-jkdch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:25:49.269803 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:49.269782    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7690-139959310141456"]
Jul 24 19:25:49.366298 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:49.366223    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkdch\" (UniqueName: \"kubernetes.io/projected/e20af6e0-f6f0-42df-8bba-67681dbc4724-kube-api-access-jkdch\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:25:50.080490 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:50.080461    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7690-139959310141456"
Jul 24 19:25:50.080490 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:50.080489    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f38cac2ddf02ee3131fe8f9c8b795dd080ee3d7c1960820cbd90e69817f196b"
Jul 24 19:25:50.090010 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:25:50.089984    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7690-139959310141456\" is forbidden: User \"system:node:ip-10-0-128-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-111.ec2.internal' and this object" podUID="e20af6e0-f6f0-42df-8bba-67681dbc4724" pod="opendatahub/test-curl-7690-139959310141456"
Jul 24 19:25:51.195323 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:25:51.195285    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7690-139959310141456\" is forbidden: User \"system:node:ip-10-0-128-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-111.ec2.internal' and this object" podUID="e20af6e0-f6f0-42df-8bba-67681dbc4724" pod="opendatahub/test-curl-7690-139959310141456"
Jul 24 19:25:51.197620 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:51.197593    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e20af6e0-f6f0-42df-8bba-67681dbc4724" path="/var/lib/kubelet/pods/e20af6e0-f6f0-42df-8bba-67681dbc4724/volumes"
Jul 24 19:25:59.163458 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.163425    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"]
Jul 24 19:25:59.164092 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.164072    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20af6e0-f6f0-42df-8bba-67681dbc4724" containerName="test-curl-7690-139959310141456"
Jul 24 19:25:59.164175 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.164094    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20af6e0-f6f0-42df-8bba-67681dbc4724" containerName="test-curl-7690-139959310141456"
Jul 24 19:25:59.164238 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.164226    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20af6e0-f6f0-42df-8bba-67681dbc4724" containerName="test-curl-7690-139959310141456"
Jul 24 19:25:59.167627 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.167606    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:25:59.170416 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.170397    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2d5d53-serving-cert\""
Jul 24 19:25:59.175849 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.175819    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"]
Jul 24 19:25:59.250846 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.250818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f980784-8537-46be-bc7a-57b98cfadf3f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9\" (UID: \"6f980784-8537-46be-bc7a-57b98cfadf3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:25:59.251004 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.250929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq57n\" (UniqueName: \"kubernetes.io/projected/6f980784-8537-46be-bc7a-57b98cfadf3f-kube-api-access-mq57n\") pod \"maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9\" (UID: \"6f980784-8537-46be-bc7a-57b98cfadf3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:25:59.251081 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.251024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f980784-8537-46be-bc7a-57b98cfadf3f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9\" (UID: \"6f980784-8537-46be-bc7a-57b98cfadf3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:25:59.352398 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.352353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq57n\" (UniqueName: \"kubernetes.io/projected/6f980784-8537-46be-bc7a-57b98cfadf3f-kube-api-access-mq57n\") pod \"maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9\" (UID: \"6f980784-8537-46be-bc7a-57b98cfadf3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:25:59.352538 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.352418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f980784-8537-46be-bc7a-57b98cfadf3f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9\" (UID: \"6f980784-8537-46be-bc7a-57b98cfadf3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:25:59.352538 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.352468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f980784-8537-46be-bc7a-57b98cfadf3f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9\" (UID: \"6f980784-8537-46be-bc7a-57b98cfadf3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:25:59.353063 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.353041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f980784-8537-46be-bc7a-57b98cfadf3f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9\" (UID: \"6f980784-8537-46be-bc7a-57b98cfadf3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:25:59.354641 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.354620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f980784-8537-46be-bc7a-57b98cfadf3f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9\" (UID: \"6f980784-8537-46be-bc7a-57b98cfadf3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:25:59.364257 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.364235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq57n\" (UniqueName: \"kubernetes.io/projected/6f980784-8537-46be-bc7a-57b98cfadf3f-kube-api-access-mq57n\") pod \"maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9\" (UID: \"6f980784-8537-46be-bc7a-57b98cfadf3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:25:59.479499 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.479425    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:25:59.621588 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:25:59.621557    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f980784_8537_46be_bc7a_57b98cfadf3f.slice/crio-c07bc26f01bdefda76ade830d20af4082bec50ae780ff0844ddaf3704950e9cf WatchSource:0}: Error finding container c07bc26f01bdefda76ade830d20af4082bec50ae780ff0844ddaf3704950e9cf: Status 404 returned error can't find the container with id c07bc26f01bdefda76ade830d20af4082bec50ae780ff0844ddaf3704950e9cf
Jul 24 19:25:59.621879 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:25:59.621853    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"]
Jul 24 19:26:00.001868 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.001839    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"]
Jul 24 19:26:00.006583 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.006563    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"
Jul 24 19:26:00.009804 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.009786    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2d5d53-dockercfg-swgpn\""
Jul 24 19:26:00.009804 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.009785    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2d5d53\""
Jul 24 19:26:00.014259 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.014232    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"]
Jul 24 19:26:00.035139 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.035114    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"]
Jul 24 19:26:00.038551 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.038533    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"
Jul 24 19:26:00.050320 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.050301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"]
Jul 24 19:26:00.062879 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.062859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghvlm\" (UniqueName: \"kubernetes.io/projected/40b4531f-21af-43aa-9caa-27a9282c4797-kube-api-access-ghvlm\") pod \"payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl\" (UID: \"40b4531f-21af-43aa-9caa-27a9282c4797\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"
Jul 24 19:26:00.062981 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.062914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b4531f-21af-43aa-9caa-27a9282c4797-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl\" (UID: \"40b4531f-21af-43aa-9caa-27a9282c4797\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"
Jul 24 19:26:00.130147 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.130118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9" event={"ID":"6f980784-8537-46be-bc7a-57b98cfadf3f","Type":"ContainerStarted","Data":"c07bc26f01bdefda76ade830d20af4082bec50ae780ff0844ddaf3704950e9cf"}
Jul 24 19:26:00.163768 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.163709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760e11bb-2ea7-4880-89f9-2a38e33894cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2\" (UID: \"760e11bb-2ea7-4880-89f9-2a38e33894cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"
Jul 24 19:26:00.164132 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.163898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghvlm\" (UniqueName: \"kubernetes.io/projected/40b4531f-21af-43aa-9caa-27a9282c4797-kube-api-access-ghvlm\") pod \"payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl\" (UID: \"40b4531f-21af-43aa-9caa-27a9282c4797\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"
Jul 24 19:26:00.164132 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.163978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx9b6\" (UniqueName: \"kubernetes.io/projected/760e11bb-2ea7-4880-89f9-2a38e33894cf-kube-api-access-nx9b6\") pod \"payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2\" (UID: \"760e11bb-2ea7-4880-89f9-2a38e33894cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"
Jul 24 19:26:00.164132 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.164051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b4531f-21af-43aa-9caa-27a9282c4797-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl\" (UID: \"40b4531f-21af-43aa-9caa-27a9282c4797\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"
Jul 24 19:26:00.164788 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.164767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b4531f-21af-43aa-9caa-27a9282c4797-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl\" (UID: \"40b4531f-21af-43aa-9caa-27a9282c4797\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"
Jul 24 19:26:00.173300 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.173279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghvlm\" (UniqueName: \"kubernetes.io/projected/40b4531f-21af-43aa-9caa-27a9282c4797-kube-api-access-ghvlm\") pod \"payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl\" (UID: \"40b4531f-21af-43aa-9caa-27a9282c4797\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"
Jul 24 19:26:00.265129 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.265038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx9b6\" (UniqueName: \"kubernetes.io/projected/760e11bb-2ea7-4880-89f9-2a38e33894cf-kube-api-access-nx9b6\") pod \"payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2\" (UID: \"760e11bb-2ea7-4880-89f9-2a38e33894cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"
Jul 24 19:26:00.265458 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.265423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760e11bb-2ea7-4880-89f9-2a38e33894cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2\" (UID: \"760e11bb-2ea7-4880-89f9-2a38e33894cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"
Jul 24 19:26:00.266114 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.266094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760e11bb-2ea7-4880-89f9-2a38e33894cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2\" (UID: \"760e11bb-2ea7-4880-89f9-2a38e33894cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"
Jul 24 19:26:00.272845 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.272817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx9b6\" (UniqueName: \"kubernetes.io/projected/760e11bb-2ea7-4880-89f9-2a38e33894cf-kube-api-access-nx9b6\") pod \"payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2\" (UID: \"760e11bb-2ea7-4880-89f9-2a38e33894cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"
Jul 24 19:26:00.318124 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.318081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"
Jul 24 19:26:00.352257 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.352228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"
Jul 24 19:26:00.514162 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.514136    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"]
Jul 24 19:26:00.515939 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:26:00.515869    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760e11bb_2ea7_4880_89f9_2a38e33894cf.slice/crio-4fb9295e31a1d0d957eb6398055cccc52aca555de3e0f493138d681826bf5f7e WatchSource:0}: Error finding container 4fb9295e31a1d0d957eb6398055cccc52aca555de3e0f493138d681826bf5f7e: Status 404 returned error can't find the container with id 4fb9295e31a1d0d957eb6398055cccc52aca555de3e0f493138d681826bf5f7e
Jul 24 19:26:00.671214 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:00.671187    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"]
Jul 24 19:26:00.673455 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:26:00.673428    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b4531f_21af_43aa_9caa_27a9282c4797.slice/crio-91b3c569558fb751efbd0f8aaa53e5d81ef62555c484b388daee638a4533facb WatchSource:0}: Error finding container 91b3c569558fb751efbd0f8aaa53e5d81ef62555c484b388daee638a4533facb: Status 404 returned error can't find the container with id 91b3c569558fb751efbd0f8aaa53e5d81ef62555c484b388daee638a4533facb
Jul 24 19:26:01.138208 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:01.138149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl" event={"ID":"40b4531f-21af-43aa-9caa-27a9282c4797","Type":"ContainerStarted","Data":"0bf9c0a88613f5b3ac1c713636c416f7d65572edfd3cba2866966e06d2bb83cf"}
Jul 24 19:26:01.138208 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:01.138213    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"
Jul 24 19:26:01.138485 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:01.138241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl" event={"ID":"40b4531f-21af-43aa-9caa-27a9282c4797","Type":"ContainerStarted","Data":"91b3c569558fb751efbd0f8aaa53e5d81ef62555c484b388daee638a4533facb"}
Jul 24 19:26:01.139969 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:01.139940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9" event={"ID":"6f980784-8537-46be-bc7a-57b98cfadf3f","Type":"ContainerStarted","Data":"c976a6c3e6598c15ac5871f1e3eaa0cbdb213bd8139f8920dec07886f050422e"}
Jul 24 19:26:01.140112 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:01.140009    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:26:01.141615 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:01.141589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2" event={"ID":"760e11bb-2ea7-4880-89f9-2a38e33894cf","Type":"ContainerStarted","Data":"44ab03c13255beeecb4ce5910426a6fc101895ba673d5d03d3a16e22907b896f"}
Jul 24 19:26:01.141774 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:01.141636    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2" event={"ID":"760e11bb-2ea7-4880-89f9-2a38e33894cf","Type":"ContainerStarted","Data":"4fb9295e31a1d0d957eb6398055cccc52aca555de3e0f493138d681826bf5f7e"}
Jul 24 19:26:01.141774 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:01.141673    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"
Jul 24 19:26:01.154166 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:01.154124    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl" podStartSLOduration=2.154111642 podStartE2EDuration="2.154111642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:01.152588702 +0000 UTC m=+2760.528247998" watchObservedRunningTime="2026-07-24 19:26:01.154111642 +0000 UTC m=+2760.529770941"
Jul 24 19:26:01.168391 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:01.168335    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2" podStartSLOduration=1.168321358 podStartE2EDuration="1.168321358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:01.166718174 +0000 UTC m=+2760.542377469" watchObservedRunningTime="2026-07-24 19:26:01.168321358 +0000 UTC m=+2760.543980653"
Jul 24 19:26:01.182786 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:01.182749    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9" podStartSLOduration=1.681999819 podStartE2EDuration="2.182734543s" totalImagesPullingTime="500.734724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:01.181820084 +0000 UTC m=+2760.557479382" watchObservedRunningTime="2026-07-24 19:26:01.182734543 +0000 UTC m=+2760.558393838"
Jul 24 19:26:07.152024 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:07.151993    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:26:12.148632 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:12.148600    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"
Jul 24 19:26:12.149075 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:12.148676    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"
Jul 24 19:26:18.392188 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:18.392139    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"]
Jul 24 19:26:18.398888 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:18.398860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:26:18.401041 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:18.401016    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8a409a-serving-cert\""
Jul 24 19:26:18.405749 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:18.405620    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"]
Jul 24 19:26:18.537976 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:18.537947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72mst\" (UniqueName: \"kubernetes.io/projected/a6629d64-8372-46c4-b000-09fa7178a08c-kube-api-access-72mst\") pod \"maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl\" (UID: \"a6629d64-8372-46c4-b000-09fa7178a08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:26:18.538125 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:18.538008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6629d64-8372-46c4-b000-09fa7178a08c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl\" (UID: \"a6629d64-8372-46c4-b000-09fa7178a08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:26:18.538125 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:18.538070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6629d64-8372-46c4-b000-09fa7178a08c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl\" (UID: \"a6629d64-8372-46c4-b000-09fa7178a08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:26:18.638935 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:18.638903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6629d64-8372-46c4-b000-09fa7178a08c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl\" (UID: \"a6629d64-8372-46c4-b000-09fa7178a08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:26:18.639090 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:18.638950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6629d64-8372-46c4-b000-09fa7178a08c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl\" (UID: \"a6629d64-8372-46c4-b000-09fa7178a08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:26:18.639090 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:18.639019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72mst\" (UniqueName: \"kubernetes.io/projected/a6629d64-8372-46c4-b000-09fa7178a08c-kube-api-access-72mst\") pod \"maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl\" (UID: \"a6629d64-8372-46c4-b000-09fa7178a08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:26:18.639090 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:26:18.639036    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-serving-cert: secret "maas-api-e2e-ipp-b-8a409a-serving-cert" not found
Jul 24 19:26:18.639090 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:26:18.639088    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6629d64-8372-46c4-b000-09fa7178a08c-maas-api-tls podName:a6629d64-8372-46c4-b000-09fa7178a08c nodeName:}" failed. No retries permitted until 2026-07-24 19:26:19.139073541 +0000 UTC m=+2778.514732828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6629d64-8372-46c4-b000-09fa7178a08c-maas-api-tls") pod "maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl" (UID: "a6629d64-8372-46c4-b000-09fa7178a08c") : secret "maas-api-e2e-ipp-b-8a409a-serving-cert" not found
Jul 24 19:26:18.639659 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:18.639641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6629d64-8372-46c4-b000-09fa7178a08c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl\" (UID: \"a6629d64-8372-46c4-b000-09fa7178a08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:26:18.648421 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:18.648348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72mst\" (UniqueName: \"kubernetes.io/projected/a6629d64-8372-46c4-b000-09fa7178a08c-kube-api-access-72mst\") pod \"maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl\" (UID: \"a6629d64-8372-46c4-b000-09fa7178a08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:26:19.053900 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.053818    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"]
Jul 24 19:26:19.058262 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.058239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"
Jul 24 19:26:19.060220 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.060194    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8a409a\""
Jul 24 19:26:19.060330 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.060205    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8a409a-dockercfg-dbg2h\""
Jul 24 19:26:19.067229 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.067209    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"]
Jul 24 19:26:19.143002 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.142969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d907e40f-04da-4f9b-91e6-7b11d1b82394-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2\" (UID: \"d907e40f-04da-4f9b-91e6-7b11d1b82394\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"
Jul 24 19:26:19.143164 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.143052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6629d64-8372-46c4-b000-09fa7178a08c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl\" (UID: \"a6629d64-8372-46c4-b000-09fa7178a08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:26:19.143164 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.143128    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2rng\" (UniqueName: \"kubernetes.io/projected/d907e40f-04da-4f9b-91e6-7b11d1b82394-kube-api-access-f2rng\") pod \"payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2\" (UID: \"d907e40f-04da-4f9b-91e6-7b11d1b82394\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"
Jul 24 19:26:19.145584 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.145552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6629d64-8372-46c4-b000-09fa7178a08c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl\" (UID: \"a6629d64-8372-46c4-b000-09fa7178a08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:26:19.244527 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.244492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2rng\" (UniqueName: \"kubernetes.io/projected/d907e40f-04da-4f9b-91e6-7b11d1b82394-kube-api-access-f2rng\") pod \"payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2\" (UID: \"d907e40f-04da-4f9b-91e6-7b11d1b82394\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"
Jul 24 19:26:19.244690 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.244586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d907e40f-04da-4f9b-91e6-7b11d1b82394-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2\" (UID: \"d907e40f-04da-4f9b-91e6-7b11d1b82394\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"
Jul 24 19:26:19.245262 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.245236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d907e40f-04da-4f9b-91e6-7b11d1b82394-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2\" (UID: \"d907e40f-04da-4f9b-91e6-7b11d1b82394\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"
Jul 24 19:26:19.253784 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.253760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2rng\" (UniqueName: \"kubernetes.io/projected/d907e40f-04da-4f9b-91e6-7b11d1b82394-kube-api-access-f2rng\") pod \"payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2\" (UID: \"d907e40f-04da-4f9b-91e6-7b11d1b82394\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"
Jul 24 19:26:19.312364 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.312304    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:26:19.371045 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.371016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"
Jul 24 19:26:19.461526 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.461497    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"]
Jul 24 19:26:19.464047 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:26:19.464021    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6629d64_8372_46c4_b000_09fa7178a08c.slice/crio-d22fc5e255751255a5c0f013e5d596b90f2306257e250d5f32bf93dd9a5301f5 WatchSource:0}: Error finding container d22fc5e255751255a5c0f013e5d596b90f2306257e250d5f32bf93dd9a5301f5: Status 404 returned error can't find the container with id d22fc5e255751255a5c0f013e5d596b90f2306257e250d5f32bf93dd9a5301f5
Jul 24 19:26:19.519986 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:19.519962    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"]
Jul 24 19:26:19.522435 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:26:19.522405    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd907e40f_04da_4f9b_91e6_7b11d1b82394.slice/crio-3faec1aaa4b0bf70dc17db6f7693e733d450e01ce7e9922b11a5e3b1b4a7f056 WatchSource:0}: Error finding container 3faec1aaa4b0bf70dc17db6f7693e733d450e01ce7e9922b11a5e3b1b4a7f056: Status 404 returned error can't find the container with id 3faec1aaa4b0bf70dc17db6f7693e733d450e01ce7e9922b11a5e3b1b4a7f056
Jul 24 19:26:20.242009 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:20.241911    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2" event={"ID":"d907e40f-04da-4f9b-91e6-7b11d1b82394","Type":"ContainerStarted","Data":"9af6f09e6282c36dc2d0152d46a7c90343109a5d6f0d6bbdfc5e2d0e008bf6d4"}
Jul 24 19:26:20.242009 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:20.241971    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2" event={"ID":"d907e40f-04da-4f9b-91e6-7b11d1b82394","Type":"ContainerStarted","Data":"3faec1aaa4b0bf70dc17db6f7693e733d450e01ce7e9922b11a5e3b1b4a7f056"}
Jul 24 19:26:20.242248 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:20.242018    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"
Jul 24 19:26:20.243546 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:20.243519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl" event={"ID":"a6629d64-8372-46c4-b000-09fa7178a08c","Type":"ContainerStarted","Data":"80cd1fed0bf4f68f07cb0880db7ef6b237042e2673f217ea6c5842266117f47e"}
Jul 24 19:26:20.243656 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:20.243561    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl" event={"ID":"a6629d64-8372-46c4-b000-09fa7178a08c","Type":"ContainerStarted","Data":"d22fc5e255751255a5c0f013e5d596b90f2306257e250d5f32bf93dd9a5301f5"}
Jul 24 19:26:20.243656 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:20.243627    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:26:20.259828 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:20.259786    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2" podStartSLOduration=1.2597696840000001 podStartE2EDuration="1.259769684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:20.256977492 +0000 UTC m=+2779.632636792" watchObservedRunningTime="2026-07-24 19:26:20.259769684 +0000 UTC m=+2779.635428981"
Jul 24 19:26:20.271967 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:20.271916    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl" podStartSLOduration=1.903990826 podStartE2EDuration="2.271901975s" totalImagesPullingTime="367.911149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:26:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:26:20.271162299 +0000 UTC m=+2779.646821596" watchObservedRunningTime="2026-07-24 19:26:20.271901975 +0000 UTC m=+2779.647561272"
Jul 24 19:26:26.255134 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:26.255051    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:26:31.251808 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:31.251777    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"
Jul 24 19:26:40.006704 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:40.006653    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pz5wb_02328292-4ed8-4caf-a880-594f6920e321/payload-processing/0.log"
Jul 24 19:26:40.163692 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:26:40.163649    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2_d907e40f-04da-4f9b-91e6-7b11d1b82394/payload-processing/0.log"
Jul 24 19:27:21.777369 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:21.777324    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2_760e11bb-2ea7-4880-89f9-2a38e33894cf/payload-processing/0.log"
Jul 24 19:27:21.897575 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:21.897541    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pz5wb_02328292-4ed8-4caf-a880-594f6920e321/payload-processing/0.log"
Jul 24 19:27:22.014660 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:22.014631    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2_d907e40f-04da-4f9b-91e6-7b11d1b82394/payload-processing/0.log"
Jul 24 19:27:25.837668 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:25.837629    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"]
Jul 24 19:27:25.841504 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:25.841486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:25.844571 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:25.844551    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-9cf26b-serving-cert\""
Jul 24 19:27:25.851455 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:25.851428    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"]
Jul 24 19:27:25.911657 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:25.911625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnrbm\" (UniqueName: \"kubernetes.io/projected/cc83fc0a-6582-4a9a-a60b-fed4a836f890-kube-api-access-dnrbm\") pod \"maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g\" (UID: \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:25.911812 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:25.911661    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc83fc0a-6582-4a9a-a60b-fed4a836f890-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g\" (UID: \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:25.911812 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:25.911722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc83fc0a-6582-4a9a-a60b-fed4a836f890-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g\" (UID: \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:26.012991 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:26.012959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnrbm\" (UniqueName: \"kubernetes.io/projected/cc83fc0a-6582-4a9a-a60b-fed4a836f890-kube-api-access-dnrbm\") pod \"maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g\" (UID: \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:26.012991 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:26.012998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc83fc0a-6582-4a9a-a60b-fed4a836f890-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g\" (UID: \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:26.013623 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:26.013043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc83fc0a-6582-4a9a-a60b-fed4a836f890-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g\" (UID: \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:26.013909 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:26.013880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc83fc0a-6582-4a9a-a60b-fed4a836f890-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g\" (UID: \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:26.015586 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:26.015564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc83fc0a-6582-4a9a-a60b-fed4a836f890-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g\" (UID: \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:26.020812 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:26.020792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnrbm\" (UniqueName: \"kubernetes.io/projected/cc83fc0a-6582-4a9a-a60b-fed4a836f890-kube-api-access-dnrbm\") pod \"maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g\" (UID: \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:26.154054 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:26.153978    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:26.291749 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:26.291708    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"]
Jul 24 19:27:26.295026 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:27:26.294999    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc83fc0a_6582_4a9a_a60b_fed4a836f890.slice/crio-13108be5c14401edb4d9c2ff238fc315fba815b6606770e91baf29a50894c1aa WatchSource:0}: Error finding container 13108be5c14401edb4d9c2ff238fc315fba815b6606770e91baf29a50894c1aa: Status 404 returned error can't find the container with id 13108be5c14401edb4d9c2ff238fc315fba815b6606770e91baf29a50894c1aa
Jul 24 19:27:26.584092 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:26.584059    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g" event={"ID":"cc83fc0a-6582-4a9a-a60b-fed4a836f890","Type":"ContainerStarted","Data":"13108be5c14401edb4d9c2ff238fc315fba815b6606770e91baf29a50894c1aa"}
Jul 24 19:27:27.590460 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:27.590418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g" event={"ID":"cc83fc0a-6582-4a9a-a60b-fed4a836f890","Type":"ContainerStarted","Data":"6e4069f92ac545e74f42c1b0c57fc237c409eda115325b2483667c72bf49e026"}
Jul 24 19:27:27.590969 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:27.590510    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:27.608540 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:27.608493    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g" podStartSLOduration=2.241138342 podStartE2EDuration="2.608479995s" totalImagesPullingTime="367.341653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:27.606088284 +0000 UTC m=+2846.981747580" watchObservedRunningTime="2026-07-24 19:27:27.608479995 +0000 UTC m=+2846.984139290"
Jul 24 19:27:33.603784 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:33.603755    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:51.576617 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:51.576580    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"]
Jul 24 19:27:51.577241 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:51.576823    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g" podUID="cc83fc0a-6582-4a9a-a60b-fed4a836f890" containerName="maas-api" containerID="cri-o://6e4069f92ac545e74f42c1b0c57fc237c409eda115325b2483667c72bf49e026" gracePeriod=30
Jul 24 19:27:51.714531 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:51.714498    2580 generic.go:356] "Generic (PLEG): container finished" podID="cc83fc0a-6582-4a9a-a60b-fed4a836f890" containerID="6e4069f92ac545e74f42c1b0c57fc237c409eda115325b2483667c72bf49e026" exitCode=0
Jul 24 19:27:51.714697 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:51.714585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g" event={"ID":"cc83fc0a-6582-4a9a-a60b-fed4a836f890","Type":"ContainerDied","Data":"6e4069f92ac545e74f42c1b0c57fc237c409eda115325b2483667c72bf49e026"}
Jul 24 19:27:51.837658 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:51.837599    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:51.945675 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:51.945643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc83fc0a-6582-4a9a-a60b-fed4a836f890-openshift-service-ca\") pod \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\" (UID: \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\") "
Jul 24 19:27:51.945836 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:51.945697    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc83fc0a-6582-4a9a-a60b-fed4a836f890-maas-api-tls\") pod \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\" (UID: \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\") "
Jul 24 19:27:51.945836 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:51.945773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnrbm\" (UniqueName: \"kubernetes.io/projected/cc83fc0a-6582-4a9a-a60b-fed4a836f890-kube-api-access-dnrbm\") pod \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\" (UID: \"cc83fc0a-6582-4a9a-a60b-fed4a836f890\") "
Jul 24 19:27:51.946046 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:51.945999    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc83fc0a-6582-4a9a-a60b-fed4a836f890-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc83fc0a-6582-4a9a-a60b-fed4a836f890" (UID: "cc83fc0a-6582-4a9a-a60b-fed4a836f890"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:27:51.946141 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:51.946123    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc83fc0a-6582-4a9a-a60b-fed4a836f890-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:27:51.947872 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:51.947843    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc83fc0a-6582-4a9a-a60b-fed4a836f890-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc83fc0a-6582-4a9a-a60b-fed4a836f890" (UID: "cc83fc0a-6582-4a9a-a60b-fed4a836f890"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:27:51.947958 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:51.947890    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc83fc0a-6582-4a9a-a60b-fed4a836f890-kube-api-access-dnrbm" (OuterVolumeSpecName: "kube-api-access-dnrbm") pod "cc83fc0a-6582-4a9a-a60b-fed4a836f890" (UID: "cc83fc0a-6582-4a9a-a60b-fed4a836f890"). InnerVolumeSpecName "kube-api-access-dnrbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:27:52.046904 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:52.046876    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnrbm\" (UniqueName: \"kubernetes.io/projected/cc83fc0a-6582-4a9a-a60b-fed4a836f890-kube-api-access-dnrbm\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:27:52.046904 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:52.046904    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc83fc0a-6582-4a9a-a60b-fed4a836f890-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:27:52.720960 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:52.720922    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"
Jul 24 19:27:52.721482 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:52.720959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g" event={"ID":"cc83fc0a-6582-4a9a-a60b-fed4a836f890","Type":"ContainerDied","Data":"13108be5c14401edb4d9c2ff238fc315fba815b6606770e91baf29a50894c1aa"}
Jul 24 19:27:52.721482 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:52.721020    2580 scope.go:117] "RemoveContainer" containerID="6e4069f92ac545e74f42c1b0c57fc237c409eda115325b2483667c72bf49e026"
Jul 24 19:27:52.744395 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:52.744349    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"]
Jul 24 19:27:52.750426 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:52.750398    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9cf26b-684bb7f8dc-g8r5g"]
Jul 24 19:27:53.197644 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:27:53.197609    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc83fc0a-6582-4a9a-a60b-fed4a836f890" path="/var/lib/kubelet/pods/cc83fc0a-6582-4a9a-a60b-fed4a836f890/volumes"
Jul 24 19:28:21.459061 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.459013    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"]
Jul 24 19:28:21.459714 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.459262    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9" podUID="6f980784-8537-46be-bc7a-57b98cfadf3f" containerName="maas-api" containerID="cri-o://c976a6c3e6598c15ac5871f1e3eaa0cbdb213bd8139f8920dec07886f050422e" gracePeriod=30
Jul 24 19:28:21.525705 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.525670    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"]
Jul 24 19:28:21.525928 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.525900    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2" podUID="760e11bb-2ea7-4880-89f9-2a38e33894cf" containerName="payload-processing" containerID="cri-o://44ab03c13255beeecb4ce5910426a6fc101895ba673d5d03d3a16e22907b896f" gracePeriod=30
Jul 24 19:28:21.532462 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.532432    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"]
Jul 24 19:28:21.532701 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.532656    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl" podUID="40b4531f-21af-43aa-9caa-27a9282c4797" containerName="payload-pre-processing" containerID="cri-o://0bf9c0a88613f5b3ac1c713636c416f7d65572edfd3cba2866966e06d2bb83cf" gracePeriod=30
Jul 24 19:28:21.792840 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.792813    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"
Jul 24 19:28:21.826913 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.826889    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:28:21.839447 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.839428    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"
Jul 24 19:28:21.877433 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.877375    2580 generic.go:356] "Generic (PLEG): container finished" podID="6f980784-8537-46be-bc7a-57b98cfadf3f" containerID="c976a6c3e6598c15ac5871f1e3eaa0cbdb213bd8139f8920dec07886f050422e" exitCode=0
Jul 24 19:28:21.877433 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.877425    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"
Jul 24 19:28:21.877670 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.877464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9" event={"ID":"6f980784-8537-46be-bc7a-57b98cfadf3f","Type":"ContainerDied","Data":"c976a6c3e6598c15ac5871f1e3eaa0cbdb213bd8139f8920dec07886f050422e"}
Jul 24 19:28:21.877670 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.877510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9" event={"ID":"6f980784-8537-46be-bc7a-57b98cfadf3f","Type":"ContainerDied","Data":"c07bc26f01bdefda76ade830d20af4082bec50ae780ff0844ddaf3704950e9cf"}
Jul 24 19:28:21.877670 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.877541    2580 scope.go:117] "RemoveContainer" containerID="c976a6c3e6598c15ac5871f1e3eaa0cbdb213bd8139f8920dec07886f050422e"
Jul 24 19:28:21.879039 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.879015    2580 generic.go:356] "Generic (PLEG): container finished" podID="760e11bb-2ea7-4880-89f9-2a38e33894cf" containerID="44ab03c13255beeecb4ce5910426a6fc101895ba673d5d03d3a16e22907b896f" exitCode=0
Jul 24 19:28:21.879153 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.879042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2" event={"ID":"760e11bb-2ea7-4880-89f9-2a38e33894cf","Type":"ContainerDied","Data":"44ab03c13255beeecb4ce5910426a6fc101895ba673d5d03d3a16e22907b896f"}
Jul 24 19:28:21.879153 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.879100    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2" event={"ID":"760e11bb-2ea7-4880-89f9-2a38e33894cf","Type":"ContainerDied","Data":"4fb9295e31a1d0d957eb6398055cccc52aca555de3e0f493138d681826bf5f7e"}
Jul 24 19:28:21.879338 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.879318    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"
Jul 24 19:28:21.880497 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.880473    2580 generic.go:356] "Generic (PLEG): container finished" podID="40b4531f-21af-43aa-9caa-27a9282c4797" containerID="0bf9c0a88613f5b3ac1c713636c416f7d65572edfd3cba2866966e06d2bb83cf" exitCode=0
Jul 24 19:28:21.880625 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.880526    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"
Jul 24 19:28:21.880625 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.880546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl" event={"ID":"40b4531f-21af-43aa-9caa-27a9282c4797","Type":"ContainerDied","Data":"0bf9c0a88613f5b3ac1c713636c416f7d65572edfd3cba2866966e06d2bb83cf"}
Jul 24 19:28:21.880625 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.880586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl" event={"ID":"40b4531f-21af-43aa-9caa-27a9282c4797","Type":"ContainerDied","Data":"91b3c569558fb751efbd0f8aaa53e5d81ef62555c484b388daee638a4533facb"}
Jul 24 19:28:21.887986 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.887970    2580 scope.go:117] "RemoveContainer" containerID="c976a6c3e6598c15ac5871f1e3eaa0cbdb213bd8139f8920dec07886f050422e"
Jul 24 19:28:21.888275 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:28:21.888256    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c976a6c3e6598c15ac5871f1e3eaa0cbdb213bd8139f8920dec07886f050422e\": container with ID starting with c976a6c3e6598c15ac5871f1e3eaa0cbdb213bd8139f8920dec07886f050422e not found: ID does not exist" containerID="c976a6c3e6598c15ac5871f1e3eaa0cbdb213bd8139f8920dec07886f050422e"
Jul 24 19:28:21.888509 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.888283    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c976a6c3e6598c15ac5871f1e3eaa0cbdb213bd8139f8920dec07886f050422e"} err="failed to get container status \"c976a6c3e6598c15ac5871f1e3eaa0cbdb213bd8139f8920dec07886f050422e\": rpc error: code = NotFound desc = could not find container \"c976a6c3e6598c15ac5871f1e3eaa0cbdb213bd8139f8920dec07886f050422e\": container with ID starting with c976a6c3e6598c15ac5871f1e3eaa0cbdb213bd8139f8920dec07886f050422e not found: ID does not exist"
Jul 24 19:28:21.888509 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.888300    2580 scope.go:117] "RemoveContainer" containerID="44ab03c13255beeecb4ce5910426a6fc101895ba673d5d03d3a16e22907b896f"
Jul 24 19:28:21.896037 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.896021    2580 scope.go:117] "RemoveContainer" containerID="44ab03c13255beeecb4ce5910426a6fc101895ba673d5d03d3a16e22907b896f"
Jul 24 19:28:21.896283 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:28:21.896258    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ab03c13255beeecb4ce5910426a6fc101895ba673d5d03d3a16e22907b896f\": container with ID starting with 44ab03c13255beeecb4ce5910426a6fc101895ba673d5d03d3a16e22907b896f not found: ID does not exist" containerID="44ab03c13255beeecb4ce5910426a6fc101895ba673d5d03d3a16e22907b896f"
Jul 24 19:28:21.896407 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.896287    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ab03c13255beeecb4ce5910426a6fc101895ba673d5d03d3a16e22907b896f"} err="failed to get container status \"44ab03c13255beeecb4ce5910426a6fc101895ba673d5d03d3a16e22907b896f\": rpc error: code = NotFound desc = could not find container \"44ab03c13255beeecb4ce5910426a6fc101895ba673d5d03d3a16e22907b896f\": container with ID starting with 44ab03c13255beeecb4ce5910426a6fc101895ba673d5d03d3a16e22907b896f not found: ID does not exist"
Jul 24 19:28:21.896407 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.896304    2580 scope.go:117] "RemoveContainer" containerID="0bf9c0a88613f5b3ac1c713636c416f7d65572edfd3cba2866966e06d2bb83cf"
Jul 24 19:28:21.903693 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.903677    2580 scope.go:117] "RemoveContainer" containerID="0bf9c0a88613f5b3ac1c713636c416f7d65572edfd3cba2866966e06d2bb83cf"
Jul 24 19:28:21.903909 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:28:21.903890    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf9c0a88613f5b3ac1c713636c416f7d65572edfd3cba2866966e06d2bb83cf\": container with ID starting with 0bf9c0a88613f5b3ac1c713636c416f7d65572edfd3cba2866966e06d2bb83cf not found: ID does not exist" containerID="0bf9c0a88613f5b3ac1c713636c416f7d65572edfd3cba2866966e06d2bb83cf"
Jul 24 19:28:21.903957 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.903914    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf9c0a88613f5b3ac1c713636c416f7d65572edfd3cba2866966e06d2bb83cf"} err="failed to get container status \"0bf9c0a88613f5b3ac1c713636c416f7d65572edfd3cba2866966e06d2bb83cf\": rpc error: code = NotFound desc = could not find container \"0bf9c0a88613f5b3ac1c713636c416f7d65572edfd3cba2866966e06d2bb83cf\": container with ID starting with 0bf9c0a88613f5b3ac1c713636c416f7d65572edfd3cba2866966e06d2bb83cf not found: ID does not exist"
Jul 24 19:28:21.908212 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.908194    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f980784-8537-46be-bc7a-57b98cfadf3f-maas-api-tls\") pod \"6f980784-8537-46be-bc7a-57b98cfadf3f\" (UID: \"6f980784-8537-46be-bc7a-57b98cfadf3f\") "
Jul 24 19:28:21.908260 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.908250    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx9b6\" (UniqueName: \"kubernetes.io/projected/760e11bb-2ea7-4880-89f9-2a38e33894cf-kube-api-access-nx9b6\") pod \"760e11bb-2ea7-4880-89f9-2a38e33894cf\" (UID: \"760e11bb-2ea7-4880-89f9-2a38e33894cf\") "
Jul 24 19:28:21.908295 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.908285    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghvlm\" (UniqueName: \"kubernetes.io/projected/40b4531f-21af-43aa-9caa-27a9282c4797-kube-api-access-ghvlm\") pod \"40b4531f-21af-43aa-9caa-27a9282c4797\" (UID: \"40b4531f-21af-43aa-9caa-27a9282c4797\") "
Jul 24 19:28:21.908338 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.908320    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq57n\" (UniqueName: \"kubernetes.io/projected/6f980784-8537-46be-bc7a-57b98cfadf3f-kube-api-access-mq57n\") pod \"6f980784-8537-46be-bc7a-57b98cfadf3f\" (UID: \"6f980784-8537-46be-bc7a-57b98cfadf3f\") "
Jul 24 19:28:21.908374 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.908346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b4531f-21af-43aa-9caa-27a9282c4797-plugins-config-volume\") pod \"40b4531f-21af-43aa-9caa-27a9282c4797\" (UID: \"40b4531f-21af-43aa-9caa-27a9282c4797\") "
Jul 24 19:28:21.908453 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.908373    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760e11bb-2ea7-4880-89f9-2a38e33894cf-plugins-config-volume\") pod \"760e11bb-2ea7-4880-89f9-2a38e33894cf\" (UID: \"760e11bb-2ea7-4880-89f9-2a38e33894cf\") "
Jul 24 19:28:21.908453 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.908410    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f980784-8537-46be-bc7a-57b98cfadf3f-openshift-service-ca\") pod \"6f980784-8537-46be-bc7a-57b98cfadf3f\" (UID: \"6f980784-8537-46be-bc7a-57b98cfadf3f\") "
Jul 24 19:28:21.908782 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.908759    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760e11bb-2ea7-4880-89f9-2a38e33894cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "760e11bb-2ea7-4880-89f9-2a38e33894cf" (UID: "760e11bb-2ea7-4880-89f9-2a38e33894cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:28:21.908892 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.908801    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b4531f-21af-43aa-9caa-27a9282c4797-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40b4531f-21af-43aa-9caa-27a9282c4797" (UID: "40b4531f-21af-43aa-9caa-27a9282c4797"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:28:21.909028 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.909004    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f980784-8537-46be-bc7a-57b98cfadf3f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f980784-8537-46be-bc7a-57b98cfadf3f" (UID: "6f980784-8537-46be-bc7a-57b98cfadf3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:28:21.910271 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.910246    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760e11bb-2ea7-4880-89f9-2a38e33894cf-kube-api-access-nx9b6" (OuterVolumeSpecName: "kube-api-access-nx9b6") pod "760e11bb-2ea7-4880-89f9-2a38e33894cf" (UID: "760e11bb-2ea7-4880-89f9-2a38e33894cf"). InnerVolumeSpecName "kube-api-access-nx9b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:28:21.910476 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.910444    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f980784-8537-46be-bc7a-57b98cfadf3f-kube-api-access-mq57n" (OuterVolumeSpecName: "kube-api-access-mq57n") pod "6f980784-8537-46be-bc7a-57b98cfadf3f" (UID: "6f980784-8537-46be-bc7a-57b98cfadf3f"). InnerVolumeSpecName "kube-api-access-mq57n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:28:21.910740 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.910723    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f980784-8537-46be-bc7a-57b98cfadf3f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f980784-8537-46be-bc7a-57b98cfadf3f" (UID: "6f980784-8537-46be-bc7a-57b98cfadf3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:28:21.910740 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:21.910732    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b4531f-21af-43aa-9caa-27a9282c4797-kube-api-access-ghvlm" (OuterVolumeSpecName: "kube-api-access-ghvlm") pod "40b4531f-21af-43aa-9caa-27a9282c4797" (UID: "40b4531f-21af-43aa-9caa-27a9282c4797"). InnerVolumeSpecName "kube-api-access-ghvlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:28:22.010098 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:22.010036    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx9b6\" (UniqueName: \"kubernetes.io/projected/760e11bb-2ea7-4880-89f9-2a38e33894cf-kube-api-access-nx9b6\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:28:22.010098 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:22.010061    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghvlm\" (UniqueName: \"kubernetes.io/projected/40b4531f-21af-43aa-9caa-27a9282c4797-kube-api-access-ghvlm\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:28:22.010098 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:22.010072    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq57n\" (UniqueName: \"kubernetes.io/projected/6f980784-8537-46be-bc7a-57b98cfadf3f-kube-api-access-mq57n\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:28:22.010098 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:22.010080    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b4531f-21af-43aa-9caa-27a9282c4797-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:28:22.010098 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:22.010089    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760e11bb-2ea7-4880-89f9-2a38e33894cf-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:28:22.010098 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:22.010098    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f980784-8537-46be-bc7a-57b98cfadf3f-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:28:22.010424 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:22.010106    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f980784-8537-46be-bc7a-57b98cfadf3f-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:28:22.205299 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:22.205274    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"]
Jul 24 19:28:22.211741 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:22.211718    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d5d53-797c8db59c-sn6z9"]
Jul 24 19:28:22.220189 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:22.220167    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"]
Jul 24 19:28:22.223624 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:22.223594    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d5d53-fbd69d4d8-zfrr2"]
Jul 24 19:28:22.232523 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:22.232502    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"]
Jul 24 19:28:22.236527 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:22.236509    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d5d53-5b5c85d668-vcrfl"]
Jul 24 19:28:23.197983 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:23.197950    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40b4531f-21af-43aa-9caa-27a9282c4797" path="/var/lib/kubelet/pods/40b4531f-21af-43aa-9caa-27a9282c4797/volumes"
Jul 24 19:28:23.200602 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:23.198316    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f980784-8537-46be-bc7a-57b98cfadf3f" path="/var/lib/kubelet/pods/6f980784-8537-46be-bc7a-57b98cfadf3f/volumes"
Jul 24 19:28:23.200602 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:28:23.198645    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="760e11bb-2ea7-4880-89f9-2a38e33894cf" path="/var/lib/kubelet/pods/760e11bb-2ea7-4880-89f9-2a38e33894cf/volumes"
Jul 24 19:29:40.242188 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.242139    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"]
Jul 24 19:29:40.242742 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.242523    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl" podUID="a6629d64-8372-46c4-b000-09fa7178a08c" containerName="maas-api" containerID="cri-o://80cd1fed0bf4f68f07cb0880db7ef6b237042e2673f217ea6c5842266117f47e" gracePeriod=30
Jul 24 19:29:40.304159 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.304121    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"]
Jul 24 19:29:40.304393 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.304350    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2" podUID="d907e40f-04da-4f9b-91e6-7b11d1b82394" containerName="payload-processing" containerID="cri-o://9af6f09e6282c36dc2d0152d46a7c90343109a5d6f0d6bbdfc5e2d0e008bf6d4" gracePeriod=30
Jul 24 19:29:40.557399 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.557364    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:29:40.583209 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.583180    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"
Jul 24 19:29:40.666803 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.666777    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d907e40f-04da-4f9b-91e6-7b11d1b82394-plugins-config-volume\") pod \"d907e40f-04da-4f9b-91e6-7b11d1b82394\" (UID: \"d907e40f-04da-4f9b-91e6-7b11d1b82394\") "
Jul 24 19:29:40.666974 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.666845    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2rng\" (UniqueName: \"kubernetes.io/projected/d907e40f-04da-4f9b-91e6-7b11d1b82394-kube-api-access-f2rng\") pod \"d907e40f-04da-4f9b-91e6-7b11d1b82394\" (UID: \"d907e40f-04da-4f9b-91e6-7b11d1b82394\") "
Jul 24 19:29:40.666974 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.666934    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72mst\" (UniqueName: \"kubernetes.io/projected/a6629d64-8372-46c4-b000-09fa7178a08c-kube-api-access-72mst\") pod \"a6629d64-8372-46c4-b000-09fa7178a08c\" (UID: \"a6629d64-8372-46c4-b000-09fa7178a08c\") "
Jul 24 19:29:40.667100 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.666974    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6629d64-8372-46c4-b000-09fa7178a08c-maas-api-tls\") pod \"a6629d64-8372-46c4-b000-09fa7178a08c\" (UID: \"a6629d64-8372-46c4-b000-09fa7178a08c\") "
Jul 24 19:29:40.667100 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.667003    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6629d64-8372-46c4-b000-09fa7178a08c-openshift-service-ca\") pod \"a6629d64-8372-46c4-b000-09fa7178a08c\" (UID: \"a6629d64-8372-46c4-b000-09fa7178a08c\") "
Jul 24 19:29:40.667364 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.667213    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d907e40f-04da-4f9b-91e6-7b11d1b82394-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d907e40f-04da-4f9b-91e6-7b11d1b82394" (UID: "d907e40f-04da-4f9b-91e6-7b11d1b82394"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:29:40.667496 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.667407    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6629d64-8372-46c4-b000-09fa7178a08c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a6629d64-8372-46c4-b000-09fa7178a08c" (UID: "a6629d64-8372-46c4-b000-09fa7178a08c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:29:40.668924 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.668901    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d907e40f-04da-4f9b-91e6-7b11d1b82394-kube-api-access-f2rng" (OuterVolumeSpecName: "kube-api-access-f2rng") pod "d907e40f-04da-4f9b-91e6-7b11d1b82394" (UID: "d907e40f-04da-4f9b-91e6-7b11d1b82394"). InnerVolumeSpecName "kube-api-access-f2rng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:29:40.669432 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.669404    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6629d64-8372-46c4-b000-09fa7178a08c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a6629d64-8372-46c4-b000-09fa7178a08c" (UID: "a6629d64-8372-46c4-b000-09fa7178a08c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:29:40.669522 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.669412    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6629d64-8372-46c4-b000-09fa7178a08c-kube-api-access-72mst" (OuterVolumeSpecName: "kube-api-access-72mst") pod "a6629d64-8372-46c4-b000-09fa7178a08c" (UID: "a6629d64-8372-46c4-b000-09fa7178a08c"). InnerVolumeSpecName "kube-api-access-72mst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:29:40.768212 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.768146    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d907e40f-04da-4f9b-91e6-7b11d1b82394-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:29:40.768212 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.768179    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2rng\" (UniqueName: \"kubernetes.io/projected/d907e40f-04da-4f9b-91e6-7b11d1b82394-kube-api-access-f2rng\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:29:40.768212 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.768189    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72mst\" (UniqueName: \"kubernetes.io/projected/a6629d64-8372-46c4-b000-09fa7178a08c-kube-api-access-72mst\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:29:40.768212 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.768198    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6629d64-8372-46c4-b000-09fa7178a08c-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:29:40.768212 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:40.768207    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6629d64-8372-46c4-b000-09fa7178a08c-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:29:41.272991 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.272952    2580 generic.go:356] "Generic (PLEG): container finished" podID="a6629d64-8372-46c4-b000-09fa7178a08c" containerID="80cd1fed0bf4f68f07cb0880db7ef6b237042e2673f217ea6c5842266117f47e" exitCode=0
Jul 24 19:29:41.273353 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.272996    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"
Jul 24 19:29:41.273353 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.273025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl" event={"ID":"a6629d64-8372-46c4-b000-09fa7178a08c","Type":"ContainerDied","Data":"80cd1fed0bf4f68f07cb0880db7ef6b237042e2673f217ea6c5842266117f47e"}
Jul 24 19:29:41.273353 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.273085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl" event={"ID":"a6629d64-8372-46c4-b000-09fa7178a08c","Type":"ContainerDied","Data":"d22fc5e255751255a5c0f013e5d596b90f2306257e250d5f32bf93dd9a5301f5"}
Jul 24 19:29:41.273353 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.273111    2580 scope.go:117] "RemoveContainer" containerID="80cd1fed0bf4f68f07cb0880db7ef6b237042e2673f217ea6c5842266117f47e"
Jul 24 19:29:41.274439 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.274412    2580 generic.go:356] "Generic (PLEG): container finished" podID="d907e40f-04da-4f9b-91e6-7b11d1b82394" containerID="9af6f09e6282c36dc2d0152d46a7c90343109a5d6f0d6bbdfc5e2d0e008bf6d4" exitCode=0
Jul 24 19:29:41.274510 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.274470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2" event={"ID":"d907e40f-04da-4f9b-91e6-7b11d1b82394","Type":"ContainerDied","Data":"9af6f09e6282c36dc2d0152d46a7c90343109a5d6f0d6bbdfc5e2d0e008bf6d4"}
Jul 24 19:29:41.274557 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.274510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2" event={"ID":"d907e40f-04da-4f9b-91e6-7b11d1b82394","Type":"ContainerDied","Data":"3faec1aaa4b0bf70dc17db6f7693e733d450e01ce7e9922b11a5e3b1b4a7f056"}
Jul 24 19:29:41.274557 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.274511    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"
Jul 24 19:29:41.282114 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.282098    2580 scope.go:117] "RemoveContainer" containerID="80cd1fed0bf4f68f07cb0880db7ef6b237042e2673f217ea6c5842266117f47e"
Jul 24 19:29:41.282358 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:29:41.282341    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80cd1fed0bf4f68f07cb0880db7ef6b237042e2673f217ea6c5842266117f47e\": container with ID starting with 80cd1fed0bf4f68f07cb0880db7ef6b237042e2673f217ea6c5842266117f47e not found: ID does not exist" containerID="80cd1fed0bf4f68f07cb0880db7ef6b237042e2673f217ea6c5842266117f47e"
Jul 24 19:29:41.282429 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.282367    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80cd1fed0bf4f68f07cb0880db7ef6b237042e2673f217ea6c5842266117f47e"} err="failed to get container status \"80cd1fed0bf4f68f07cb0880db7ef6b237042e2673f217ea6c5842266117f47e\": rpc error: code = NotFound desc = could not find container \"80cd1fed0bf4f68f07cb0880db7ef6b237042e2673f217ea6c5842266117f47e\": container with ID starting with 80cd1fed0bf4f68f07cb0880db7ef6b237042e2673f217ea6c5842266117f47e not found: ID does not exist"
Jul 24 19:29:41.282429 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.282400    2580 scope.go:117] "RemoveContainer" containerID="9af6f09e6282c36dc2d0152d46a7c90343109a5d6f0d6bbdfc5e2d0e008bf6d4"
Jul 24 19:29:41.289903 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.289884    2580 scope.go:117] "RemoveContainer" containerID="9af6f09e6282c36dc2d0152d46a7c90343109a5d6f0d6bbdfc5e2d0e008bf6d4"
Jul 24 19:29:41.290145 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:29:41.290126    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af6f09e6282c36dc2d0152d46a7c90343109a5d6f0d6bbdfc5e2d0e008bf6d4\": container with ID starting with 9af6f09e6282c36dc2d0152d46a7c90343109a5d6f0d6bbdfc5e2d0e008bf6d4 not found: ID does not exist" containerID="9af6f09e6282c36dc2d0152d46a7c90343109a5d6f0d6bbdfc5e2d0e008bf6d4"
Jul 24 19:29:41.290209 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.290153    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af6f09e6282c36dc2d0152d46a7c90343109a5d6f0d6bbdfc5e2d0e008bf6d4"} err="failed to get container status \"9af6f09e6282c36dc2d0152d46a7c90343109a5d6f0d6bbdfc5e2d0e008bf6d4\": rpc error: code = NotFound desc = could not find container \"9af6f09e6282c36dc2d0152d46a7c90343109a5d6f0d6bbdfc5e2d0e008bf6d4\": container with ID starting with 9af6f09e6282c36dc2d0152d46a7c90343109a5d6f0d6bbdfc5e2d0e008bf6d4 not found: ID does not exist"
Jul 24 19:29:41.299979 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.299959    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"]
Jul 24 19:29:41.310136 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.310114    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a409a-555b5b6688-lprsl"]
Jul 24 19:29:41.325958 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.325937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"]
Jul 24 19:29:41.331326 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:41.331306    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a409a-644bb5c4bb-zjfn2"]
Jul 24 19:29:43.197357 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:43.197322    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6629d64-8372-46c4-b000-09fa7178a08c" path="/var/lib/kubelet/pods/a6629d64-8372-46c4-b000-09fa7178a08c/volumes"
Jul 24 19:29:43.197744 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:29:43.197703    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d907e40f-04da-4f9b-91e6-7b11d1b82394" path="/var/lib/kubelet/pods/d907e40f-04da-4f9b-91e6-7b11d1b82394/volumes"
Jul 24 19:30:01.328946 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:01.328919    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:30:01.333549 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:01.333524    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:30:02.336342 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.336311    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"]
Jul 24 19:30:02.336709 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.336516    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm" podUID="ee47147c-3f74-46d1-9b47-0a7062cc0db6" containerName="maas-api" containerID="cri-o://40b07e1fe045603f80e951f29534401b113a82428389255ac92e789fa6811a9f" gracePeriod=30
Jul 24 19:30:02.365752 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.365725    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm" podUID="ee47147c-3f74-46d1-9b47-0a7062cc0db6" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.69:8443/health\": dial tcp 10.132.0.69:8443: connect: connection refused"
Jul 24 19:30:02.391361 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.391336    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"]
Jul 24 19:30:02.391628 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.391588    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8" podUID="0690cb74-7813-4c14-895c-f1d6cde543db" containerName="payload-processing" containerID="cri-o://c96071a68194b97733cbe8ae76fe02a41dba238b3569f737147db437220529f6" gracePeriod=30
Jul 24 19:30:02.398402 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.398351    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"]
Jul 24 19:30:02.399190 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.398565    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm" podUID="04ffc07c-c370-461f-9d6f-39d1e4b563f8" containerName="payload-pre-processing" containerID="cri-o://3f9e4ebcdaa0c3e1c073791d6cd5a6d6ead9a60af3c0a65b55e85b5661e4ff6a" gracePeriod=30
Jul 24 19:30:02.642786 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.642753    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:30:02.686238 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.686214    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"
Jul 24 19:30:02.696974 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.696949    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"
Jul 24 19:30:02.755204 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.755176    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee47147c-3f74-46d1-9b47-0a7062cc0db6-maas-api-tls\") pod \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\" (UID: \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\") "
Jul 24 19:30:02.755395 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.755288    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv6pr\" (UniqueName: \"kubernetes.io/projected/ee47147c-3f74-46d1-9b47-0a7062cc0db6-kube-api-access-sv6pr\") pod \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\" (UID: \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\") "
Jul 24 19:30:02.755395 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.755341    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee47147c-3f74-46d1-9b47-0a7062cc0db6-openshift-service-ca\") pod \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\" (UID: \"ee47147c-3f74-46d1-9b47-0a7062cc0db6\") "
Jul 24 19:30:02.755780 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.755749    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee47147c-3f74-46d1-9b47-0a7062cc0db6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee47147c-3f74-46d1-9b47-0a7062cc0db6" (UID: "ee47147c-3f74-46d1-9b47-0a7062cc0db6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:30:02.757338 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.757312    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee47147c-3f74-46d1-9b47-0a7062cc0db6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee47147c-3f74-46d1-9b47-0a7062cc0db6" (UID: "ee47147c-3f74-46d1-9b47-0a7062cc0db6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:30:02.757459 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.757348    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee47147c-3f74-46d1-9b47-0a7062cc0db6-kube-api-access-sv6pr" (OuterVolumeSpecName: "kube-api-access-sv6pr") pod "ee47147c-3f74-46d1-9b47-0a7062cc0db6" (UID: "ee47147c-3f74-46d1-9b47-0a7062cc0db6"). InnerVolumeSpecName "kube-api-access-sv6pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:30:02.856664 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.856597    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ffc07c-c370-461f-9d6f-39d1e4b563f8-plugins-config-volume\") pod \"04ffc07c-c370-461f-9d6f-39d1e4b563f8\" (UID: \"04ffc07c-c370-461f-9d6f-39d1e4b563f8\") "
Jul 24 19:30:02.856664 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.856635    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0690cb74-7813-4c14-895c-f1d6cde543db-plugins-config-volume\") pod \"0690cb74-7813-4c14-895c-f1d6cde543db\" (UID: \"0690cb74-7813-4c14-895c-f1d6cde543db\") "
Jul 24 19:30:02.856864 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.856670    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbm4f\" (UniqueName: \"kubernetes.io/projected/0690cb74-7813-4c14-895c-f1d6cde543db-kube-api-access-nbm4f\") pod \"0690cb74-7813-4c14-895c-f1d6cde543db\" (UID: \"0690cb74-7813-4c14-895c-f1d6cde543db\") "
Jul 24 19:30:02.856864 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.856711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x72zv\" (UniqueName: \"kubernetes.io/projected/04ffc07c-c370-461f-9d6f-39d1e4b563f8-kube-api-access-x72zv\") pod \"04ffc07c-c370-461f-9d6f-39d1e4b563f8\" (UID: \"04ffc07c-c370-461f-9d6f-39d1e4b563f8\") "
Jul 24 19:30:02.856979 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.856905    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee47147c-3f74-46d1-9b47-0a7062cc0db6-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:30:02.856979 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.856917    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv6pr\" (UniqueName: \"kubernetes.io/projected/ee47147c-3f74-46d1-9b47-0a7062cc0db6-kube-api-access-sv6pr\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:30:02.856979 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.856927    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee47147c-3f74-46d1-9b47-0a7062cc0db6-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:30:02.857119 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.857029    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ffc07c-c370-461f-9d6f-39d1e4b563f8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "04ffc07c-c370-461f-9d6f-39d1e4b563f8" (UID: "04ffc07c-c370-461f-9d6f-39d1e4b563f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:30:02.857119 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.857035    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0690cb74-7813-4c14-895c-f1d6cde543db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0690cb74-7813-4c14-895c-f1d6cde543db" (UID: "0690cb74-7813-4c14-895c-f1d6cde543db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:30:02.858672 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.858648    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0690cb74-7813-4c14-895c-f1d6cde543db-kube-api-access-nbm4f" (OuterVolumeSpecName: "kube-api-access-nbm4f") pod "0690cb74-7813-4c14-895c-f1d6cde543db" (UID: "0690cb74-7813-4c14-895c-f1d6cde543db"). InnerVolumeSpecName "kube-api-access-nbm4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:30:02.858756 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.858739    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ffc07c-c370-461f-9d6f-39d1e4b563f8-kube-api-access-x72zv" (OuterVolumeSpecName: "kube-api-access-x72zv") pod "04ffc07c-c370-461f-9d6f-39d1e4b563f8" (UID: "04ffc07c-c370-461f-9d6f-39d1e4b563f8"). InnerVolumeSpecName "kube-api-access-x72zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:30:02.957731 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.957703    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ffc07c-c370-461f-9d6f-39d1e4b563f8-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:30:02.957731 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.957729    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0690cb74-7813-4c14-895c-f1d6cde543db-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:30:02.957902 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.957738    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbm4f\" (UniqueName: \"kubernetes.io/projected/0690cb74-7813-4c14-895c-f1d6cde543db-kube-api-access-nbm4f\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:30:02.957902 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:02.957747    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x72zv\" (UniqueName: \"kubernetes.io/projected/04ffc07c-c370-461f-9d6f-39d1e4b563f8-kube-api-access-x72zv\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:30:03.389170 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.389137    2580 generic.go:356] "Generic (PLEG): container finished" podID="04ffc07c-c370-461f-9d6f-39d1e4b563f8" containerID="3f9e4ebcdaa0c3e1c073791d6cd5a6d6ead9a60af3c0a65b55e85b5661e4ff6a" exitCode=0
Jul 24 19:30:03.389615 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.389175    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"
Jul 24 19:30:03.389615 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.389228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm" event={"ID":"04ffc07c-c370-461f-9d6f-39d1e4b563f8","Type":"ContainerDied","Data":"3f9e4ebcdaa0c3e1c073791d6cd5a6d6ead9a60af3c0a65b55e85b5661e4ff6a"}
Jul 24 19:30:03.389615 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.389280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm" event={"ID":"04ffc07c-c370-461f-9d6f-39d1e4b563f8","Type":"ContainerDied","Data":"1345f3b3dc03f95192e040585ccd7b4393caa2ad60a0a7bf00a3f4451657097f"}
Jul 24 19:30:03.389615 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.389302    2580 scope.go:117] "RemoveContainer" containerID="3f9e4ebcdaa0c3e1c073791d6cd5a6d6ead9a60af3c0a65b55e85b5661e4ff6a"
Jul 24 19:30:03.390608 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.390588    2580 generic.go:356] "Generic (PLEG): container finished" podID="0690cb74-7813-4c14-895c-f1d6cde543db" containerID="c96071a68194b97733cbe8ae76fe02a41dba238b3569f737147db437220529f6" exitCode=0
Jul 24 19:30:03.390717 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.390621    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"
Jul 24 19:30:03.390717 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.390698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8" event={"ID":"0690cb74-7813-4c14-895c-f1d6cde543db","Type":"ContainerDied","Data":"c96071a68194b97733cbe8ae76fe02a41dba238b3569f737147db437220529f6"}
Jul 24 19:30:03.390832 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.390748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8" event={"ID":"0690cb74-7813-4c14-895c-f1d6cde543db","Type":"ContainerDied","Data":"37ee1e9a711d764657abe31f83e021891997c5af9775105bbe69f452669b5904"}
Jul 24 19:30:03.392468 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.392446    2580 generic.go:356] "Generic (PLEG): container finished" podID="ee47147c-3f74-46d1-9b47-0a7062cc0db6" containerID="40b07e1fe045603f80e951f29534401b113a82428389255ac92e789fa6811a9f" exitCode=0
Jul 24 19:30:03.392563 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.392479    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm" event={"ID":"ee47147c-3f74-46d1-9b47-0a7062cc0db6","Type":"ContainerDied","Data":"40b07e1fe045603f80e951f29534401b113a82428389255ac92e789fa6811a9f"}
Jul 24 19:30:03.392563 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.392487    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"
Jul 24 19:30:03.392563 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.392519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm" event={"ID":"ee47147c-3f74-46d1-9b47-0a7062cc0db6","Type":"ContainerDied","Data":"ccc8b78fe9f6a5bd6b09c6602fd3001e740308204886004652be944aa4f1397d"}
Jul 24 19:30:03.399214 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.399195    2580 scope.go:117] "RemoveContainer" containerID="3f9e4ebcdaa0c3e1c073791d6cd5a6d6ead9a60af3c0a65b55e85b5661e4ff6a"
Jul 24 19:30:03.399503 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:30:03.399475    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9e4ebcdaa0c3e1c073791d6cd5a6d6ead9a60af3c0a65b55e85b5661e4ff6a\": container with ID starting with 3f9e4ebcdaa0c3e1c073791d6cd5a6d6ead9a60af3c0a65b55e85b5661e4ff6a not found: ID does not exist" containerID="3f9e4ebcdaa0c3e1c073791d6cd5a6d6ead9a60af3c0a65b55e85b5661e4ff6a"
Jul 24 19:30:03.399557 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.399504    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9e4ebcdaa0c3e1c073791d6cd5a6d6ead9a60af3c0a65b55e85b5661e4ff6a"} err="failed to get container status \"3f9e4ebcdaa0c3e1c073791d6cd5a6d6ead9a60af3c0a65b55e85b5661e4ff6a\": rpc error: code = NotFound desc = could not find container \"3f9e4ebcdaa0c3e1c073791d6cd5a6d6ead9a60af3c0a65b55e85b5661e4ff6a\": container with ID starting with 3f9e4ebcdaa0c3e1c073791d6cd5a6d6ead9a60af3c0a65b55e85b5661e4ff6a not found: ID does not exist"
Jul 24 19:30:03.399557 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.399520    2580 scope.go:117] "RemoveContainer" containerID="c96071a68194b97733cbe8ae76fe02a41dba238b3569f737147db437220529f6"
Jul 24 19:30:03.406869 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.406851    2580 scope.go:117] "RemoveContainer" containerID="c96071a68194b97733cbe8ae76fe02a41dba238b3569f737147db437220529f6"
Jul 24 19:30:03.407169 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:30:03.407152    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96071a68194b97733cbe8ae76fe02a41dba238b3569f737147db437220529f6\": container with ID starting with c96071a68194b97733cbe8ae76fe02a41dba238b3569f737147db437220529f6 not found: ID does not exist" containerID="c96071a68194b97733cbe8ae76fe02a41dba238b3569f737147db437220529f6"
Jul 24 19:30:03.407222 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.407174    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96071a68194b97733cbe8ae76fe02a41dba238b3569f737147db437220529f6"} err="failed to get container status \"c96071a68194b97733cbe8ae76fe02a41dba238b3569f737147db437220529f6\": rpc error: code = NotFound desc = could not find container \"c96071a68194b97733cbe8ae76fe02a41dba238b3569f737147db437220529f6\": container with ID starting with c96071a68194b97733cbe8ae76fe02a41dba238b3569f737147db437220529f6 not found: ID does not exist"
Jul 24 19:30:03.407222 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.407192    2580 scope.go:117] "RemoveContainer" containerID="40b07e1fe045603f80e951f29534401b113a82428389255ac92e789fa6811a9f"
Jul 24 19:30:03.410621 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.410597    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"]
Jul 24 19:30:03.413961 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.413935    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-65184f-74d9f8fc4b-jqvp8"]
Jul 24 19:30:03.415608 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.415587    2580 scope.go:117] "RemoveContainer" containerID="40b07e1fe045603f80e951f29534401b113a82428389255ac92e789fa6811a9f"
Jul 24 19:30:03.415853 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:30:03.415832    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b07e1fe045603f80e951f29534401b113a82428389255ac92e789fa6811a9f\": container with ID starting with 40b07e1fe045603f80e951f29534401b113a82428389255ac92e789fa6811a9f not found: ID does not exist" containerID="40b07e1fe045603f80e951f29534401b113a82428389255ac92e789fa6811a9f"
Jul 24 19:30:03.415903 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.415861    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b07e1fe045603f80e951f29534401b113a82428389255ac92e789fa6811a9f"} err="failed to get container status \"40b07e1fe045603f80e951f29534401b113a82428389255ac92e789fa6811a9f\": rpc error: code = NotFound desc = could not find container \"40b07e1fe045603f80e951f29534401b113a82428389255ac92e789fa6811a9f\": container with ID starting with 40b07e1fe045603f80e951f29534401b113a82428389255ac92e789fa6811a9f not found: ID does not exist"
Jul 24 19:30:03.424490 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.424470    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"]
Jul 24 19:30:03.428923 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.428900    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-65184f-7797479cc6-2jgrm"]
Jul 24 19:30:03.441451 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.441423    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"]
Jul 24 19:30:03.443680 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:03.443659    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-65184f-56d55d9666-njtlm"]
Jul 24 19:30:05.196456 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:05.196422    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04ffc07c-c370-461f-9d6f-39d1e4b563f8" path="/var/lib/kubelet/pods/04ffc07c-c370-461f-9d6f-39d1e4b563f8/volumes"
Jul 24 19:30:05.196835 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:05.196787    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0690cb74-7813-4c14-895c-f1d6cde543db" path="/var/lib/kubelet/pods/0690cb74-7813-4c14-895c-f1d6cde543db/volumes"
Jul 24 19:30:05.197137 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:05.197124    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee47147c-3f74-46d1-9b47-0a7062cc0db6" path="/var/lib/kubelet/pods/ee47147c-3f74-46d1-9b47-0a7062cc0db6/volumes"
Jul 24 19:30:25.346191 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.346153    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"]
Jul 24 19:30:25.346759 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.346370    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8" podUID="962db09a-a11e-4b9a-ae6f-29981e81150c" containerName="maas-api" containerID="cri-o://c60881a6f72febd8cba506d615bf5c725293146aaf1bb1aae03a78f7d94e31b5" gracePeriod=30
Jul 24 19:30:25.414489 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.414459    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"]
Jul 24 19:30:25.414739 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.414716    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx" podUID="35710028-b82f-4f57-a927-abcd5d2bd0d5" containerName="payload-processing" containerID="cri-o://f75fc127e839636950a5b044ab370b25700a78c3f1199fbb73b6e06f1eb81137" gracePeriod=30
Jul 24 19:30:25.420553 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.420273    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"]
Jul 24 19:30:25.420553 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.420512    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn" podUID="15af558e-bfc3-459c-9624-4f5cc872b03f" containerName="payload-pre-processing" containerID="cri-o://ff48e2d812d8e1077f04850e938670bcf40ac19a3e58c0271e78d64b658a0e33" gracePeriod=30
Jul 24 19:30:25.507103 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.507072    2580 generic.go:356] "Generic (PLEG): container finished" podID="962db09a-a11e-4b9a-ae6f-29981e81150c" containerID="c60881a6f72febd8cba506d615bf5c725293146aaf1bb1aae03a78f7d94e31b5" exitCode=0
Jul 24 19:30:25.507254 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.507159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8" event={"ID":"962db09a-a11e-4b9a-ae6f-29981e81150c","Type":"ContainerDied","Data":"c60881a6f72febd8cba506d615bf5c725293146aaf1bb1aae03a78f7d94e31b5"}
Jul 24 19:30:25.700837 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.700816    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"
Jul 24 19:30:25.703864 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.703834    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:30:25.711798 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.711779    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"
Jul 24 19:30:25.750499 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.750473    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jckgs\" (UniqueName: \"kubernetes.io/projected/35710028-b82f-4f57-a927-abcd5d2bd0d5-kube-api-access-jckgs\") pod \"35710028-b82f-4f57-a927-abcd5d2bd0d5\" (UID: \"35710028-b82f-4f57-a927-abcd5d2bd0d5\") "
Jul 24 19:30:25.750645 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.750509    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35710028-b82f-4f57-a927-abcd5d2bd0d5-plugins-config-volume\") pod \"35710028-b82f-4f57-a927-abcd5d2bd0d5\" (UID: \"35710028-b82f-4f57-a927-abcd5d2bd0d5\") "
Jul 24 19:30:25.750645 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.750534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/962db09a-a11e-4b9a-ae6f-29981e81150c-maas-api-tls\") pod \"962db09a-a11e-4b9a-ae6f-29981e81150c\" (UID: \"962db09a-a11e-4b9a-ae6f-29981e81150c\") "
Jul 24 19:30:25.750645 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.750592    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z22lx\" (UniqueName: \"kubernetes.io/projected/15af558e-bfc3-459c-9624-4f5cc872b03f-kube-api-access-z22lx\") pod \"15af558e-bfc3-459c-9624-4f5cc872b03f\" (UID: \"15af558e-bfc3-459c-9624-4f5cc872b03f\") "
Jul 24 19:30:25.750645 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.750623    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/962db09a-a11e-4b9a-ae6f-29981e81150c-openshift-service-ca\") pod \"962db09a-a11e-4b9a-ae6f-29981e81150c\" (UID: \"962db09a-a11e-4b9a-ae6f-29981e81150c\") "
Jul 24 19:30:25.750860 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.750657    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15af558e-bfc3-459c-9624-4f5cc872b03f-plugins-config-volume\") pod \"15af558e-bfc3-459c-9624-4f5cc872b03f\" (UID: \"15af558e-bfc3-459c-9624-4f5cc872b03f\") "
Jul 24 19:30:25.750860 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.750694    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn6hz\" (UniqueName: \"kubernetes.io/projected/962db09a-a11e-4b9a-ae6f-29981e81150c-kube-api-access-vn6hz\") pod \"962db09a-a11e-4b9a-ae6f-29981e81150c\" (UID: \"962db09a-a11e-4b9a-ae6f-29981e81150c\") "
Jul 24 19:30:25.751063 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.751035    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/962db09a-a11e-4b9a-ae6f-29981e81150c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "962db09a-a11e-4b9a-ae6f-29981e81150c" (UID: "962db09a-a11e-4b9a-ae6f-29981e81150c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:30:25.751159 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.751065    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15af558e-bfc3-459c-9624-4f5cc872b03f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15af558e-bfc3-459c-9624-4f5cc872b03f" (UID: "15af558e-bfc3-459c-9624-4f5cc872b03f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:30:25.751159 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.751097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35710028-b82f-4f57-a927-abcd5d2bd0d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "35710028-b82f-4f57-a927-abcd5d2bd0d5" (UID: "35710028-b82f-4f57-a927-abcd5d2bd0d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 19:30:25.752702 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.752671    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15af558e-bfc3-459c-9624-4f5cc872b03f-kube-api-access-z22lx" (OuterVolumeSpecName: "kube-api-access-z22lx") pod "15af558e-bfc3-459c-9624-4f5cc872b03f" (UID: "15af558e-bfc3-459c-9624-4f5cc872b03f"). InnerVolumeSpecName "kube-api-access-z22lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:30:25.752934 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.752909    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35710028-b82f-4f57-a927-abcd5d2bd0d5-kube-api-access-jckgs" (OuterVolumeSpecName: "kube-api-access-jckgs") pod "35710028-b82f-4f57-a927-abcd5d2bd0d5" (UID: "35710028-b82f-4f57-a927-abcd5d2bd0d5"). InnerVolumeSpecName "kube-api-access-jckgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:30:25.753018 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.752980    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962db09a-a11e-4b9a-ae6f-29981e81150c-kube-api-access-vn6hz" (OuterVolumeSpecName: "kube-api-access-vn6hz") pod "962db09a-a11e-4b9a-ae6f-29981e81150c" (UID: "962db09a-a11e-4b9a-ae6f-29981e81150c"). InnerVolumeSpecName "kube-api-access-vn6hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 19:30:25.753077 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.753064    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/962db09a-a11e-4b9a-ae6f-29981e81150c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "962db09a-a11e-4b9a-ae6f-29981e81150c" (UID: "962db09a-a11e-4b9a-ae6f-29981e81150c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 19:30:25.851902 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.851869    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jckgs\" (UniqueName: \"kubernetes.io/projected/35710028-b82f-4f57-a927-abcd5d2bd0d5-kube-api-access-jckgs\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:30:25.851902 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.851898    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35710028-b82f-4f57-a927-abcd5d2bd0d5-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:30:25.852093 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.851911    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/962db09a-a11e-4b9a-ae6f-29981e81150c-maas-api-tls\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:30:25.852093 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.851924    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z22lx\" (UniqueName: \"kubernetes.io/projected/15af558e-bfc3-459c-9624-4f5cc872b03f-kube-api-access-z22lx\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:30:25.852093 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.851936    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/962db09a-a11e-4b9a-ae6f-29981e81150c-openshift-service-ca\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:30:25.852093 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.851948    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15af558e-bfc3-459c-9624-4f5cc872b03f-plugins-config-volume\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:30:25.852093 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:25.851962    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn6hz\" (UniqueName: \"kubernetes.io/projected/962db09a-a11e-4b9a-ae6f-29981e81150c-kube-api-access-vn6hz\") on node \"ip-10-0-128-111.ec2.internal\" DevicePath \"\""
Jul 24 19:30:26.514175 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.514144    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"
Jul 24 19:30:26.514625 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.514191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8" event={"ID":"962db09a-a11e-4b9a-ae6f-29981e81150c","Type":"ContainerDied","Data":"6831da63e7aebf0468a6ab3d836f56a382239cb301db16323f7bb7e1b176ad63"}
Jul 24 19:30:26.514625 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.514255    2580 scope.go:117] "RemoveContainer" containerID="c60881a6f72febd8cba506d615bf5c725293146aaf1bb1aae03a78f7d94e31b5"
Jul 24 19:30:26.515681 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.515659    2580 generic.go:356] "Generic (PLEG): container finished" podID="15af558e-bfc3-459c-9624-4f5cc872b03f" containerID="ff48e2d812d8e1077f04850e938670bcf40ac19a3e58c0271e78d64b658a0e33" exitCode=0
Jul 24 19:30:26.515772 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.515694    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"
Jul 24 19:30:26.515772 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.515741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn" event={"ID":"15af558e-bfc3-459c-9624-4f5cc872b03f","Type":"ContainerDied","Data":"ff48e2d812d8e1077f04850e938670bcf40ac19a3e58c0271e78d64b658a0e33"}
Jul 24 19:30:26.515876 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.515796    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn" event={"ID":"15af558e-bfc3-459c-9624-4f5cc872b03f","Type":"ContainerDied","Data":"c735737413446ec1ac54ce9a917afeb5e0c7b7d911e82419b9ddbff23a6e02cd"}
Jul 24 19:30:26.517254 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.517225    2580 generic.go:356] "Generic (PLEG): container finished" podID="35710028-b82f-4f57-a927-abcd5d2bd0d5" containerID="f75fc127e839636950a5b044ab370b25700a78c3f1199fbb73b6e06f1eb81137" exitCode=0
Jul 24 19:30:26.517416 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.517270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx" event={"ID":"35710028-b82f-4f57-a927-abcd5d2bd0d5","Type":"ContainerDied","Data":"f75fc127e839636950a5b044ab370b25700a78c3f1199fbb73b6e06f1eb81137"}
Jul 24 19:30:26.517416 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.517282    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"
Jul 24 19:30:26.517416 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.517312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx" event={"ID":"35710028-b82f-4f57-a927-abcd5d2bd0d5","Type":"ContainerDied","Data":"ed5551cfc06d3674fbafff13587568bf4ec56857b27a5e59334cf482b61a60dc"}
Jul 24 19:30:26.523288 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.523267    2580 scope.go:117] "RemoveContainer" containerID="ff48e2d812d8e1077f04850e938670bcf40ac19a3e58c0271e78d64b658a0e33"
Jul 24 19:30:26.535284 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.535038    2580 scope.go:117] "RemoveContainer" containerID="ff48e2d812d8e1077f04850e938670bcf40ac19a3e58c0271e78d64b658a0e33"
Jul 24 19:30:26.535426 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:30:26.535405    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff48e2d812d8e1077f04850e938670bcf40ac19a3e58c0271e78d64b658a0e33\": container with ID starting with ff48e2d812d8e1077f04850e938670bcf40ac19a3e58c0271e78d64b658a0e33 not found: ID does not exist" containerID="ff48e2d812d8e1077f04850e938670bcf40ac19a3e58c0271e78d64b658a0e33"
Jul 24 19:30:26.535488 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.535434    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff48e2d812d8e1077f04850e938670bcf40ac19a3e58c0271e78d64b658a0e33"} err="failed to get container status \"ff48e2d812d8e1077f04850e938670bcf40ac19a3e58c0271e78d64b658a0e33\": rpc error: code = NotFound desc = could not find container \"ff48e2d812d8e1077f04850e938670bcf40ac19a3e58c0271e78d64b658a0e33\": container with ID starting with ff48e2d812d8e1077f04850e938670bcf40ac19a3e58c0271e78d64b658a0e33 not found: ID does not exist"
Jul 24 19:30:26.535488 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.535451    2580 scope.go:117] "RemoveContainer" containerID="f75fc127e839636950a5b044ab370b25700a78c3f1199fbb73b6e06f1eb81137"
Jul 24 19:30:26.537033 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.537014    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"]
Jul 24 19:30:26.539671 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.539631    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d391c5-8655b7cf6d-m4nz8"]
Jul 24 19:30:26.543371 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.543356    2580 scope.go:117] "RemoveContainer" containerID="f75fc127e839636950a5b044ab370b25700a78c3f1199fbb73b6e06f1eb81137"
Jul 24 19:30:26.543657 ip-10-0-128-111 kubenswrapper[2580]: E0724 19:30:26.543635    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f75fc127e839636950a5b044ab370b25700a78c3f1199fbb73b6e06f1eb81137\": container with ID starting with f75fc127e839636950a5b044ab370b25700a78c3f1199fbb73b6e06f1eb81137 not found: ID does not exist" containerID="f75fc127e839636950a5b044ab370b25700a78c3f1199fbb73b6e06f1eb81137"
Jul 24 19:30:26.543734 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.543661    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75fc127e839636950a5b044ab370b25700a78c3f1199fbb73b6e06f1eb81137"} err="failed to get container status \"f75fc127e839636950a5b044ab370b25700a78c3f1199fbb73b6e06f1eb81137\": rpc error: code = NotFound desc = could not find container \"f75fc127e839636950a5b044ab370b25700a78c3f1199fbb73b6e06f1eb81137\": container with ID starting with f75fc127e839636950a5b044ab370b25700a78c3f1199fbb73b6e06f1eb81137 not found: ID does not exist"
Jul 24 19:30:26.548797 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.548559    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"]
Jul 24 19:30:26.550823 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.550801    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d391c5-77b7c95b8d-z67nn"]
Jul 24 19:30:26.559346 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.559327    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"]
Jul 24 19:30:26.562732 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:26.562711    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d391c5-67775b986b-b5clx"]
Jul 24 19:30:27.196556 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:27.196517    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15af558e-bfc3-459c-9624-4f5cc872b03f" path="/var/lib/kubelet/pods/15af558e-bfc3-459c-9624-4f5cc872b03f/volumes"
Jul 24 19:30:27.196969 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:27.196949    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35710028-b82f-4f57-a927-abcd5d2bd0d5" path="/var/lib/kubelet/pods/35710028-b82f-4f57-a927-abcd5d2bd0d5/volumes"
Jul 24 19:30:27.197368 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:27.197350    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="962db09a-a11e-4b9a-ae6f-29981e81150c" path="/var/lib/kubelet/pods/962db09a-a11e-4b9a-ae6f-29981e81150c/volumes"
Jul 24 19:30:42.082958 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:42.082931    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64bbdb967c-t228l_513c3d5f-ad2d-4852-ad67-bdbfc750fa42/manager/0.log"
Jul 24 19:30:42.675519 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:42.675492    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-859dddff6f-km4rp_e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd/copy-manifests/0.log"
Jul 24 19:30:42.683573 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:42.683545    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-859dddff6f-km4rp_e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd/manager/0.log"
Jul 24 19:30:43.029858 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:43.029787    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bbf9886d8-kn7md_07ebc295-677f-4143-9882-a569761a46cd/maas-api/0.log"
Jul 24 19:30:43.473320 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:43.473291    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-w24m2_d79dca2c-fa23-441a-8a18-f5b6fc0ab9f5/postgres/0.log"
Jul 24 19:30:45.081649 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:45.081620    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-x5x4z_1cd9e89c-1fc9-487c-a346-9179f1e74d13/kuadrant-console-plugin/0.log"
Jul 24 19:30:46.126901 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:46.126859    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bbbd7cd9b-8wkrq_8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd/kube-auth-proxy/0.log"
Jul 24 19:30:46.480763 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:46.480679    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-w44xq_6f271893-54fc-4c88-b0dd-6b5affc95805/payload-pre-processing/0.log"
Jul 24 19:30:46.602476 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:46.602445    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pz5wb_02328292-4ed8-4caf-a880-594f6920e321/payload-processing/0.log"
Jul 24 19:30:47.048342 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:47.048291    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-wr8gf_a572efa1-e59f-4f4f-8448-3517c4bf9e05/main/0.log"
Jul 24 19:30:47.267303 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:47.267262    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-sfl6n_d1a177f9-63e2-402c-93e4-374946bd82a1/main/0.log"
Jul 24 19:30:56.176852 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:56.176778    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vn9rh_8bc4eada-1823-4bc2-add7-98ef884c80d4/global-pull-secret-syncer/0.log"
Jul 24 19:30:56.263085 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:56.263037    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6mlfh_48692fa5-2ce7-4575-bacb-e0e6c5ca40a1/konnectivity-agent/0.log"
Jul 24 19:30:56.334146 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:30:56.334114    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-111.ec2.internal_6550cfaced065c17322d923d252f38de/haproxy/0.log"
Jul 24 19:31:00.950078 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:00.950049    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-x5x4z_1cd9e89c-1fc9-487c-a346-9179f1e74d13/kuadrant-console-plugin/0.log"
Jul 24 19:31:03.179357 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.179325    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-x8dd2_b5b11062-82e6-4c0f-a8ab-b78968310aac/kube-state-metrics/0.log"
Jul 24 19:31:03.214956 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.214887    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-x8dd2_b5b11062-82e6-4c0f-a8ab-b78968310aac/kube-rbac-proxy-main/0.log"
Jul 24 19:31:03.235541 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.235516    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-x8dd2_b5b11062-82e6-4c0f-a8ab-b78968310aac/kube-rbac-proxy-self/0.log"
Jul 24 19:31:03.409990 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.409964    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6cdbt_2cb33685-a791-4ae1-8d34-e2e600caa35b/node-exporter/0.log"
Jul 24 19:31:03.430030 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.430005    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6cdbt_2cb33685-a791-4ae1-8d34-e2e600caa35b/kube-rbac-proxy/0.log"
Jul 24 19:31:03.450156 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.450138    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6cdbt_2cb33685-a791-4ae1-8d34-e2e600caa35b/init-textfile/0.log"
Jul 24 19:31:03.698270 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.698244    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43cbe10e-ddce-4634-9a4c-8264ace66023/prometheus/0.log"
Jul 24 19:31:03.722806 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.722784    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43cbe10e-ddce-4634-9a4c-8264ace66023/config-reloader/0.log"
Jul 24 19:31:03.744226 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.744201    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43cbe10e-ddce-4634-9a4c-8264ace66023/thanos-sidecar/0.log"
Jul 24 19:31:03.763948 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.763920    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43cbe10e-ddce-4634-9a4c-8264ace66023/kube-rbac-proxy-web/0.log"
Jul 24 19:31:03.782904 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.782851    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43cbe10e-ddce-4634-9a4c-8264ace66023/kube-rbac-proxy/0.log"
Jul 24 19:31:03.807334 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.807315    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43cbe10e-ddce-4634-9a4c-8264ace66023/kube-rbac-proxy-thanos/0.log"
Jul 24 19:31:03.826630 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.826608    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_43cbe10e-ddce-4634-9a4c-8264ace66023/init-config-reloader/0.log"
Jul 24 19:31:03.854941 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.854916    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-7pcfl_00540c96-49d5-4d7c-8310-d0a309848478/prometheus-operator/0.log"
Jul 24 19:31:03.872026 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.872000    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-7pcfl_00540c96-49d5-4d7c-8310-d0a309848478/kube-rbac-proxy/0.log"
Jul 24 19:31:03.926532 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.926504    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-565765d559-zdkfg_5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9/telemeter-client/0.log"
Jul 24 19:31:03.948742 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.948718    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-565765d559-zdkfg_5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9/reload/0.log"
Jul 24 19:31:03.970209 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:03.970182    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-565765d559-zdkfg_5071a7b0-f0b6-4cf9-9e3e-5d235703a6b9/kube-rbac-proxy/0.log"
Jul 24 19:31:05.342490 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:05.342458    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-kj4n8_0d66c79b-9713-4e02-a2d6-82ba72b7f08a/networking-console-plugin/0.log"
Jul 24 19:31:05.975655 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:05.975617    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/1.log"
Jul 24 19:31:05.984121 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:05.984099    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-79qbp_788bff11-3f28-40aa-b52d-caebf1949d75/console-operator/2.log"
Jul 24 19:31:06.531441 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531405    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"]
Jul 24 19:31:06.531841 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531756    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b4531f-21af-43aa-9caa-27a9282c4797" containerName="payload-pre-processing"
Jul 24 19:31:06.531841 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531767    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b4531f-21af-43aa-9caa-27a9282c4797" containerName="payload-pre-processing"
Jul 24 19:31:06.531841 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531778    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="962db09a-a11e-4b9a-ae6f-29981e81150c" containerName="maas-api"
Jul 24 19:31:06.531841 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531783    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="962db09a-a11e-4b9a-ae6f-29981e81150c" containerName="maas-api"
Jul 24 19:31:06.531841 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531790    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f980784-8537-46be-bc7a-57b98cfadf3f" containerName="maas-api"
Jul 24 19:31:06.531841 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531796    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f980784-8537-46be-bc7a-57b98cfadf3f" containerName="maas-api"
Jul 24 19:31:06.531841 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531802    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760e11bb-2ea7-4880-89f9-2a38e33894cf" containerName="payload-processing"
Jul 24 19:31:06.531841 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531823    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="760e11bb-2ea7-4880-89f9-2a38e33894cf" containerName="payload-processing"
Jul 24 19:31:06.531841 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531834    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ffc07c-c370-461f-9d6f-39d1e4b563f8" containerName="payload-pre-processing"
Jul 24 19:31:06.531841 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531839    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ffc07c-c370-461f-9d6f-39d1e4b563f8" containerName="payload-pre-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531849    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d907e40f-04da-4f9b-91e6-7b11d1b82394" containerName="payload-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531855    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d907e40f-04da-4f9b-91e6-7b11d1b82394" containerName="payload-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531862    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6629d64-8372-46c4-b000-09fa7178a08c" containerName="maas-api"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531868    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6629d64-8372-46c4-b000-09fa7178a08c" containerName="maas-api"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531874    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc83fc0a-6582-4a9a-a60b-fed4a836f890" containerName="maas-api"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531879    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc83fc0a-6582-4a9a-a60b-fed4a836f890" containerName="maas-api"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531890    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35710028-b82f-4f57-a927-abcd5d2bd0d5" containerName="payload-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531895    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="35710028-b82f-4f57-a927-abcd5d2bd0d5" containerName="payload-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531903    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee47147c-3f74-46d1-9b47-0a7062cc0db6" containerName="maas-api"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531908    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee47147c-3f74-46d1-9b47-0a7062cc0db6" containerName="maas-api"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531914    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0690cb74-7813-4c14-895c-f1d6cde543db" containerName="payload-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531919    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0690cb74-7813-4c14-895c-f1d6cde543db" containerName="payload-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531929    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15af558e-bfc3-459c-9624-4f5cc872b03f" containerName="payload-pre-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531934    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15af558e-bfc3-459c-9624-4f5cc872b03f" containerName="payload-pre-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531984    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ffc07c-c370-461f-9d6f-39d1e4b563f8" containerName="payload-pre-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531992    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="962db09a-a11e-4b9a-ae6f-29981e81150c" containerName="maas-api"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.531998    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d907e40f-04da-4f9b-91e6-7b11d1b82394" containerName="payload-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.532007    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40b4531f-21af-43aa-9caa-27a9282c4797" containerName="payload-pre-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.532012    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee47147c-3f74-46d1-9b47-0a7062cc0db6" containerName="maas-api"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.532019    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="35710028-b82f-4f57-a927-abcd5d2bd0d5" containerName="payload-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.532024    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f980784-8537-46be-bc7a-57b98cfadf3f" containerName="maas-api"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.532029    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="760e11bb-2ea7-4880-89f9-2a38e33894cf" containerName="payload-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.532043    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6629d64-8372-46c4-b000-09fa7178a08c" containerName="maas-api"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.532048    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0690cb74-7813-4c14-895c-f1d6cde543db" containerName="payload-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.532052    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15af558e-bfc3-459c-9624-4f5cc872b03f" containerName="payload-pre-processing"
Jul 24 19:31:06.532206 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.532058    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc83fc0a-6582-4a9a-a60b-fed4a836f890" containerName="maas-api"
Jul 24 19:31:06.536625 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.536607    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.538926 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.538905    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vd7mh\"/\"default-dockercfg-xcq4b\""
Jul 24 19:31:06.539049 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.538933    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vd7mh\"/\"kube-root-ca.crt\""
Jul 24 19:31:06.539367 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.539350    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vd7mh\"/\"openshift-service-ca.crt\""
Jul 24 19:31:06.544259 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.544238    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"]
Jul 24 19:31:06.593847 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.593821    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-podres\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.593997 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.593876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-sys\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.593997 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.593928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-lib-modules\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.593997 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.593958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-proc\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.594172 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.593993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw98z\" (UniqueName: \"kubernetes.io/projected/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-kube-api-access-hw98z\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.600668 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.600639    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-w8s88_491c9be2-642b-4b48-828d-e72db595bfe8/download-server/0.log"
Jul 24 19:31:06.694602 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.694581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-lib-modules\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.694722 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.694613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-proc\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.694722 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.694631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw98z\" (UniqueName: \"kubernetes.io/projected/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-kube-api-access-hw98z\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.694722 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.694692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-podres\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.694722 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.694711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-proc\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.694941 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.694747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-lib-modules\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.694941 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.694757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-sys\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.694941 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.694789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-sys\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.694941 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.694810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-podres\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.701954 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.701933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw98z\" (UniqueName: \"kubernetes.io/projected/5c37ac5f-190b-4ff8-81cd-ed1db13523bf-kube-api-access-hw98z\") pod \"perf-node-gather-daemonset-lx87z\" (UID: \"5c37ac5f-190b-4ff8-81cd-ed1db13523bf\") " pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.846632 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.846608    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:06.975018 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:06.974986    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"]
Jul 24 19:31:06.978288 ip-10-0-128-111 kubenswrapper[2580]: W0724 19:31:06.978257    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5c37ac5f_190b_4ff8_81cd_ed1db13523bf.slice/crio-3bc5989bf74dd534c6bf46d0d971fd1ad68818c5eac82a7e546f1038df6c93f3 WatchSource:0}: Error finding container 3bc5989bf74dd534c6bf46d0d971fd1ad68818c5eac82a7e546f1038df6c93f3: Status 404 returned error can't find the container with id 3bc5989bf74dd534c6bf46d0d971fd1ad68818c5eac82a7e546f1038df6c93f3
Jul 24 19:31:07.714123 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:07.714088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z" event={"ID":"5c37ac5f-190b-4ff8-81cd-ed1db13523bf","Type":"ContainerStarted","Data":"57a62af9cf900dea141d6349c3b25ea2f4d351dc9e7e91b3b75a19289bcb8bcf"}
Jul 24 19:31:07.714517 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:07.714141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z" event={"ID":"5c37ac5f-190b-4ff8-81cd-ed1db13523bf","Type":"ContainerStarted","Data":"3bc5989bf74dd534c6bf46d0d971fd1ad68818c5eac82a7e546f1038df6c93f3"}
Jul 24 19:31:07.714517 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:07.714168    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:07.730087 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:07.730046    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z" podStartSLOduration=1.730033006 podStartE2EDuration="1.730033006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 19:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:31:07.729958404 +0000 UTC m=+3067.105617724" watchObservedRunningTime="2026-07-24 19:31:07.730033006 +0000 UTC m=+3067.105692342"
Jul 24 19:31:08.553960 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:08.553930    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pfdnx_44c73487-26c9-4984-a178-1b20cf791453/dns/0.log"
Jul 24 19:31:08.573825 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:08.573802    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pfdnx_44c73487-26c9-4984-a178-1b20cf791453/kube-rbac-proxy/0.log"
Jul 24 19:31:08.653464 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:08.653439    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l9drv_28bdcc77-a763-49e4-abb3-98de9bd1d399/dns-node-resolver/0.log"
Jul 24 19:31:09.224233 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:09.224203    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9nzfl_2360ee4d-31bb-4de1-91ad-278352f88f22/node-ca/0.log"
Jul 24 19:31:10.331059 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:10.331028    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bbbd7cd9b-8wkrq_8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd/kube-auth-proxy/0.log"
Jul 24 19:31:10.480391 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:10.480359    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-w44xq_6f271893-54fc-4c88-b0dd-6b5affc95805/payload-pre-processing/0.log"
Jul 24 19:31:10.518943 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:10.518910    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pz5wb_02328292-4ed8-4caf-a880-594f6920e321/payload-processing/0.log"
Jul 24 19:31:11.319259 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:11.319228    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b2cwr_99213b47-96d3-4c66-8007-33c8b2fe15d7/serve-healthcheck-canary/0.log"
Jul 24 19:31:11.853259 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:11.853236    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k276f_04e6b58a-4a62-4635-9058-0899b5799cfe/kube-rbac-proxy/0.log"
Jul 24 19:31:11.875340 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:11.875315    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k276f_04e6b58a-4a62-4635-9058-0899b5799cfe/exporter/0.log"
Jul 24 19:31:11.895595 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:11.895567    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k276f_04e6b58a-4a62-4635-9058-0899b5799cfe/extractor/0.log"
Jul 24 19:31:13.727916 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:13.727884    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vd7mh/perf-node-gather-daemonset-lx87z"
Jul 24 19:31:14.491660 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:14.491629    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64bbdb967c-t228l_513c3d5f-ad2d-4852-ad67-bdbfc750fa42/manager/0.log"
Jul 24 19:31:14.812194 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:14.812099    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-859dddff6f-km4rp_e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd/manager/0.log"
Jul 24 19:31:14.830200 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:14.830174    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-859dddff6f-km4rp_e0b4b3d4-3ce1-48fd-98e2-e6b8383354cd/copy-manifests/0.log"
Jul 24 19:31:15.993748 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:15.993719    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8456f466bb-25cmx_a2ce7387-1f69-4770-8939-902c30a71e61/manager/0.log"
Jul 24 19:31:16.046444 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:16.046418    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-282wj_554f313d-854f-4821-803c-63c6c0fe081f/openshift-lws-operator/0.log"
Jul 24 19:31:16.539028 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:16.539001    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bbbd7cd9b-8wkrq_8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd/kube-auth-proxy/0.log"
Jul 24 19:31:16.664525 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:16.664495    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-w44xq_6f271893-54fc-4c88-b0dd-6b5affc95805/payload-pre-processing/0.log"
Jul 24 19:31:16.694623 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:16.694596    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pz5wb_02328292-4ed8-4caf-a880-594f6920e321/payload-processing/0.log"
Jul 24 19:31:18.350138 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:18.350094    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bbbd7cd9b-8wkrq_8f1bca43-c0dc-4c6c-88d7-a945c7c95bfd/kube-auth-proxy/0.log"
Jul 24 19:31:18.477951 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:18.477924    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-w44xq_6f271893-54fc-4c88-b0dd-6b5affc95805/payload-pre-processing/0.log"
Jul 24 19:31:18.506465 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:18.506435    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pz5wb_02328292-4ed8-4caf-a880-594f6920e321/payload-processing/0.log"
Jul 24 19:31:21.377661 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:21.377612    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-tnw9x_600033b2-d859-44d2-9c67-5f16e8ce6ad0/migrator/0.log"
Jul 24 19:31:21.407864 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:21.407839    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-tnw9x_600033b2-d859-44d2-9c67-5f16e8ce6ad0/graceful-termination/0.log"
Jul 24 19:31:21.866323 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:21.866291    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts_7fd19688-7aa4-4567-8ada-87afa3c0a7a9/kube-storage-version-migrator-operator/1.log"
Jul 24 19:31:21.868131 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:21.868101    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-w82ts_7fd19688-7aa4-4567-8ada-87afa3c0a7a9/kube-storage-version-migrator-operator/0.log"
Jul 24 19:31:22.903672 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:22.903641    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9wq59_1455f583-c317-4edc-98d3-9c8a7f14d37b/kube-multus-additional-cni-plugins/0.log"
Jul 24 19:31:22.925064 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:22.925040    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9wq59_1455f583-c317-4edc-98d3-9c8a7f14d37b/egress-router-binary-copy/0.log"
Jul 24 19:31:22.950979 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:22.950939    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9wq59_1455f583-c317-4edc-98d3-9c8a7f14d37b/cni-plugins/0.log"
Jul 24 19:31:22.974025 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:22.974006    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9wq59_1455f583-c317-4edc-98d3-9c8a7f14d37b/bond-cni-plugin/0.log"
Jul 24 19:31:22.996670 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:22.996647    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9wq59_1455f583-c317-4edc-98d3-9c8a7f14d37b/routeoverride-cni/0.log"
Jul 24 19:31:23.019769 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:23.019743    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9wq59_1455f583-c317-4edc-98d3-9c8a7f14d37b/whereabouts-cni-bincopy/0.log"
Jul 24 19:31:23.042227 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:23.042212    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9wq59_1455f583-c317-4edc-98d3-9c8a7f14d37b/whereabouts-cni/0.log"
Jul 24 19:31:23.442702 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:23.442670    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pnpwv_feb5428a-ec41-4135-9a40-5bff75da000b/kube-multus/0.log"
Jul 24 19:31:23.557770 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:23.557703    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xkxwb_2e0785e8-c533-44cb-bc6f-d11ed1bb60f0/network-metrics-daemon/0.log"
Jul 24 19:31:23.573126 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:23.573104    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xkxwb_2e0785e8-c533-44cb-bc6f-d11ed1bb60f0/kube-rbac-proxy/0.log"
Jul 24 19:31:24.639560 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:24.639531    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hrxnr_3e257688-dca4-4017-a362-6760fac66ee2/ovn-controller/0.log"
Jul 24 19:31:24.657371 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:24.657344    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hrxnr_3e257688-dca4-4017-a362-6760fac66ee2/ovn-acl-logging/0.log"
Jul 24 19:31:24.681040 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:24.681014    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hrxnr_3e257688-dca4-4017-a362-6760fac66ee2/kube-rbac-proxy-node/0.log"
Jul 24 19:31:24.702862 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:24.702839    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hrxnr_3e257688-dca4-4017-a362-6760fac66ee2/kube-rbac-proxy-ovn-metrics/0.log"
Jul 24 19:31:24.721956 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:24.721934    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hrxnr_3e257688-dca4-4017-a362-6760fac66ee2/northd/0.log"
Jul 24 19:31:24.740595 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:24.740570    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hrxnr_3e257688-dca4-4017-a362-6760fac66ee2/nbdb/0.log"
Jul 24 19:31:24.759117 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:24.759097    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hrxnr_3e257688-dca4-4017-a362-6760fac66ee2/sbdb/0.log"
Jul 24 19:31:24.860879 ip-10-0-128-111 kubenswrapper[2580]: I0724 19:31:24.860852    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hrxnr_3e257688-dca4-4017-a362-6760fac66ee2/ovnkube-controller/0.log"