Jul 29 19:00:33.591610 ip-10-0-135-248 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 19:00:33.943847 ip-10-0-135-248 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 19:00:33.943847 ip-10-0-135-248 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 19:00:33.943847 ip-10-0-135-248 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 19:00:33.943847 ip-10-0-135-248 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 19:00:33.943847 ip-10-0-135-248 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 19:00:33.944881 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.944736    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 19:00:33.948810 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948784    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:00:33.948810 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948805    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:00:33.948810 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948809    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:00:33.948810 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948814    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:00:33.948810 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948819    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948824    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948827    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948834    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948838    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948842    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948846    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948849    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948853    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948857    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948861    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948865    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948869    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948873    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948877    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948881    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948884    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948888    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948891    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948895    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:00:33.949124 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948899    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948904    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948908    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948912    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948938    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948942    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948945    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948950    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948954    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948957    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948962    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948966    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948970    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948974    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948981    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948989    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948993    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.948998    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949003    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949007    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:00:33.949948 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949012    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949016    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949020    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949025    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949029    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949033    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949037    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949041    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949045    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949049    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949054    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949058    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949062    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949067    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949071    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949075    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949079    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949083    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949087    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949092    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:00:33.950795 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949096    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949100    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949104    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949108    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949113    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949118    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949122    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949126    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949130    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949134    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949138    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949147    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949153    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949157    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949162    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949168    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949178    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949183    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949210    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:00:33.951808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949216    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949221    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949225    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949229    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949234    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949238    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949242    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949246    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949250    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949254    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949258    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949263    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949266    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949271    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949275    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.949281    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950500    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950511    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950518    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950523    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:00:33.952621 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950527    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950532    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950537    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950542    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950546    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950552    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950557    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950562    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950567    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950573    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950579    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950583    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950588    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950592    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950595    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950600    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950603    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950608    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950612    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950616    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:00:33.953494 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950620    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950625    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950629    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950633    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950637    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950641    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950645    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950651    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950655    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950659    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950664    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950668    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950674    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950678    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950682    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950686    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950691    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950695    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950699    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:00:33.954173 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950704    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950708    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950712    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950717    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950722    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950729    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950734    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950739    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950743    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950748    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950752    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950756    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950761    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950765    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950769    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950773    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950780    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950785    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950789    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:00:33.954831 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950794    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950799    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950803    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950808    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950812    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950817    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950822    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950826    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950832    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950837    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950841    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950846    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950851    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950855    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950859    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950864    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950869    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950874    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950879    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950883    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:00:33.955435 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950887    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950892    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950895    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950899    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950904    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950908    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950912    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950915    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950920    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950924    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950928    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950933    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950937    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950942    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950946    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950950    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.950954    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951627    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951636    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951641    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:00:33.955940 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951645    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951649    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951653    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951658    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951662    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951666    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951670    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951674    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951678    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951684    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951688    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951692    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951695    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951699    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951704    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951708    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951712    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951716    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951720    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951725    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:00:33.956586 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951729    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951738    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951742    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951759    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951764    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951768    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951772    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951776    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951781    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951786    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951790    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951794    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951798    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951802    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951806    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951811    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951816    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951819    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951823    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951827    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:00:33.957081 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951832    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951836    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951841    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951845    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951849    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951856    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951862    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951866    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951871    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951875    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951879    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951883    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951888    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951894    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951898    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951902    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951906    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951913    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951918    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:00:33.957603 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951923    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951930    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951934    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951939    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951943    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951947    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951951    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951956    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951959    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951964    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951968    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951972    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951976    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951980    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951984    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951988    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951993    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.951998    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952001    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:00:33.958104 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952005    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952011    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952015    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952019    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952023    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952030    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952036    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952043    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952048    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952053    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952058    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952063    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952067    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952071    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952078    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952082    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952086    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.952090    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952220    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952233    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952244    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 19:00:33.958593 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952251    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952257    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952262    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952269    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952277    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952282    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952287    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952292    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952297    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952309    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952315    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952319    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952324    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952329    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952333    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952339    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952344    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952349    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952357    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952362    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952369    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952373    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952378    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 19:00:33.959105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952383    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952388    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952395    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952400    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952406    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952410    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952418    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952423    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952427    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952432    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952437    2579 flags.go:64] FLAG: --enable-server="true"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952443    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952452    2579 flags.go:64] FLAG: --event-burst="100"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952457    2579 flags.go:64] FLAG: --event-qps="50"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952461    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952466    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952471    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952478    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952482    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952488    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952492    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952497    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952502    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952506    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952512    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 19:00:33.959752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952517    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952521    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952528    2579 flags.go:64] FLAG: --feature-gates=""
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952534    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952539    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952544    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952549    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952554    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952559    2579 flags.go:64] FLAG: --help="false"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952566    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-135-248.ec2.internal"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952571    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952576    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952580    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952586    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952592    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952597    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952602    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952606    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952611    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952616    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952620    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952625    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952629    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952634    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 19:00:33.960430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952639    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952643    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952647    2579 flags.go:64] FLAG: --lock-file=""
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952653    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952658    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952663    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952673    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952677    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952682    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952687    2579 flags.go:64] FLAG: --logging-format="text"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952692    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952697    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952701    2579 flags.go:64] FLAG: --manifest-url=""
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952706    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952713    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952718    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952725    2579 flags.go:64] FLAG: --max-pods="110"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952731    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952735    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952740    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952745    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952750    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952754    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952759    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952770    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 19:00:33.961034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952775    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952780    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952785    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952790    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952799    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952804    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952808    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952814    2579 flags.go:64] FLAG: --port="10250"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952818    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952823    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0178fff64d962e1d8"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952828    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952834    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952839    2579 flags.go:64] FLAG: --register-node="true"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952844    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952850    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952854    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952859    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952863    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952870    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952876    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952880    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952885    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952890    2579 flags.go:64] FLAG: --runonce="false"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952894    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952901    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 19:00:33.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952906    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952911    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952916    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952921    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952927    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952932    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952937    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952941    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952946    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952951    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952956    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952961    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952966    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952975    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952979    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952984    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952991    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.952996    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.953000    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.953005    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.953010    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.953015    2579 flags.go:64] FLAG: --v="2"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.953021    2579 flags.go:64] FLAG: --version="false"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.953028    2579 flags.go:64] FLAG: --vmodule=""
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.953034    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 19:00:33.962362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.953040    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953273    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953283    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953289    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953294    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953299    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953305    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953310    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953314    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953318    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953322    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953327    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953331    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953335    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953340    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953344    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953348    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953352    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953356    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:00:33.963026 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953361    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953364    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953369    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953372    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953376    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953381    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953384    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953388    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953394    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953398    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953402    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953406    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953411    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953415    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953419    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953423    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953428    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953432    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953436    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953442    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:00:33.963518 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953446    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953450    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953454    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953458    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953462    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953467    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953471    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953475    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953479    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953483    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953487    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953490    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953494    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953498    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953502    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953505    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953509    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953513    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953517    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953521    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:00:33.964022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953527    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953532    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953536    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953543    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953548    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953553    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953558    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953563    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953567    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953572    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953577    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953584    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953589    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953593    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953597    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953602    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953606    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953610    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953614    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953618    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:00:33.964545 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953623    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953627    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953631    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953635    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953639    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953644    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953648    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953652    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953655    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953660    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953664    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953668    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953675    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953681    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953690    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953694    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953700    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953705    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953709    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:00:33.965030 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953713    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.953717    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.953726    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.961636    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.961656    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.961680    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.961690    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961762    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961769    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961772    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961775    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961777    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961780    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961783    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961785    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:00:33.965613 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961788    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961791    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961793    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961796    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961798    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961802    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961806    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961809    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961812    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961815    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961818    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961820    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961823    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961825    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961828    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961831    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961833    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961836    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961838    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961841    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:00:33.965982 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961843    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961857    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961860    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961862    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961865    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961868    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961870    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961873    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961875    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961878    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961881    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961883    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961886    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961888    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961891    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961894    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961896    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961898    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961901    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961903    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:00:33.966499 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961906    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961909    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961911    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961914    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961916    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961919    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961921    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961923    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961926    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961928    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961931    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961933    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961937    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961942    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961951    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961954    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961956    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961959    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:00:33.966979 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961964    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961967    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961970    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961979    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961982    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961985    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961987    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961990    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961992    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961995    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.961998    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962000    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962003    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962005    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962008    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962010    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962013    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962015    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962018    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962021    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:00:33.967442 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962023    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962026    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962028    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962030    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962033    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962035    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962038    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962041    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962044    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962052    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962055    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962057    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962060    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.962065    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962222    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:00:33.967947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962228    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962231    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962234    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962237    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962239    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962242    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962246    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962250    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962253    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962255    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962258    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962261    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962263    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962266    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962269    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962271    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962274    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962276    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962279    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:00:33.968346 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962281    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962283    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962286    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962288    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962291    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962294    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962297    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962300    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962308    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962311    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962314    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962318    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962320    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962323    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962325    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962328    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962330    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962333    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962335    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962337    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:00:33.968801 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962340    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962342    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962344    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962347    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962349    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962352    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962354    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962357    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962359    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962362    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962364    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962367    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962371    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962374    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962378    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962381    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962384    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962387    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962390    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:00:33.969401 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962393    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962395    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962403    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962406    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962408    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962411    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962413    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962416    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962418    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962421    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962423    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962426    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962428    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962430    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962433    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962435    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962438    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962440    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962443    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962445    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:00:33.969872 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962448    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962450    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962452    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962455    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962457    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962459    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962462    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962468    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962471    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962473    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962476    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962479    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962481    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962484    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962486    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962494    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962498    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962500    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962503    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:00:33.970374 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:33.962505    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:00:33.970868 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.962510    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 19:00:33.970868 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.962661    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 19:00:33.970868 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.965968    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 19:00:33.970868 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.967270    2579 server.go:1019] "Starting client certificate rotation"
Jul 29 19:00:33.970868 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.967392    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 19:00:33.970868 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.967432    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 19:00:33.970868 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.970353    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 19:00:33.975490 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.975462    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 19:00:33.985764 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.985747    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 29 19:00:33.987722 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.987709    2579 log.go:25] "Validated CRI v1 image API"
Jul 29 19:00:33.988549 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.988535    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 19:00:33.990979 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.990956    2579 fs.go:135] Filesystem UUIDs: map[02d84565-5b8a-491f-af31-f5dd32448274:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f6ac645b-3807-40c3-8654-8304b8ce9df7:/dev/nvme0n1p4]
Jul 29 19:00:33.991054 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.990978    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 19:00:33.996431 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.996315    2579 manager.go:216] Machine: {Timestamp:2026-07-29 19:00:33.994892885 +0000 UTC m=+0.294565376 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099585 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f0b17980e3b4988361cb7a9b34d65 SystemUUID:ec2f0b17-980e-3b49-8836-1cb7a9b34d65 BootID:3f3a9509-a468-4ba7-a292-11da1c588480 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:13:54:7a:f0:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:13:54:7a:f0:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:a5:3e:06:c3:28 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 19:00:33.996431 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.996428    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 19:00:33.996550 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.996524    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 19:00:33.998298 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.998281    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 19:00:33.999095 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.999063    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 19:00:33.999281 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.999101    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-248.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 19:00:33.999848 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.999838    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 19:00:33.999883 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.999851    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 19:00:33.999883 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:33.999859    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 19:00:34.001006 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.000994    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 19:00:34.002366 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.002356    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 19:00:34.006066 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.006054    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 19:00:34.006224 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.006212    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 19:00:34.006262 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.006252    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 19:00:34.006291 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.006268    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 19:00:34.006291 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.006289    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 19:00:34.006344 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.006305    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 29 19:00:34.006344 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.006319    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 19:00:34.006902 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.006887    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 19:00:34.007015 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.006911    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 19:00:34.009270 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.009252    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 19:00:34.010610 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.010589    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 19:00:34.010652 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.010626    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 19:00:34.011280 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.011267    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 19:00:34.011321 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.011291    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 19:00:34.011321 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.011297    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 19:00:34.011321 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.011302    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 19:00:34.011321 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.011312    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 19:00:34.011321 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.011317    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 19:00:34.011447 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.011326    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 19:00:34.011447 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.011333    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 19:00:34.011447 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.011339    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 19:00:34.011447 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.011345    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 19:00:34.011447 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.011354    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 19:00:34.011447 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.011359    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 19:00:34.011998 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.011987    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 19:00:34.012029 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.012001    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 19:00:34.014403 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.014380    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vfxwg"
Jul 29 19:00:34.014728 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.014708    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 19:00:34.014809 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.014796    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-248.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 19:00:34.014865 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.014833    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 19:00:34.015947 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.015933    2579 server.go:1268] "Started kubelet"
Jul 29 19:00:34.016060 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.016000    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 19:00:34.019847 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.016185    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 19:00:34.019977 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.019959    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 19:00:34.020724 ip-10-0-135-248 systemd[1]: Started Kubernetes Kubelet.
Jul 29 19:00:34.023432 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.021879    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vfxwg"
Jul 29 19:00:34.023964 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.023943    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 19:00:34.024597 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.024555    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 19:00:34.025979 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.025093    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-248.ec2.internal.18c6d7da18ae5718  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-248.ec2.internal,UID:ip-10-0-135-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-248.ec2.internal,},FirstTimestamp:2026-07-29 19:00:34.015901464 +0000 UTC m=+0.315573956,LastTimestamp:2026-07-29 19:00:34.015901464 +0000 UTC m=+0.315573956,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-248.ec2.internal,}"
Jul 29 19:00:34.026208 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.026169    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 19:00:34.031002 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.030986    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 19:00:34.031077 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.031003    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 19:00:34.031291 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.031262    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found"
Jul 29 19:00:34.031740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.031725    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 19:00:34.031840 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.031759    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 19:00:34.031840 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.031723    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 19:00:34.031956 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.031911    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 19:00:34.031956 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.031919    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 19:00:34.032396 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.032365    2579 factory.go:55] Registering systemd factory
Jul 29 19:00:34.032396 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.032390    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 29 19:00:34.032680 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.032657    2579 factory.go:153] Registering CRI-O factory
Jul 29 19:00:34.032680 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.032676    2579 factory.go:223] Registration of the crio container factory successfully
Jul 29 19:00:34.032809 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.032733    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 19:00:34.032809 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.032756    2579 factory.go:103] Registering Raw factory
Jul 29 19:00:34.032809 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.032770    2579 manager.go:1196] Started watching for new ooms in manager
Jul 29 19:00:34.033145 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.033134    2579 manager.go:349] Starting recovery of all containers
Jul 29 19:00:34.034121 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.033787    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 19:00:34.034121 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.033884    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 19:00:34.044860 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.044845    2579 manager.go:354] Recovery completed
Jul 29 19:00:34.049552 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.049534    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:00:34.051684 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.051669    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:00:34.051758 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.051697    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:00:34.051758 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.051707    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:00:34.052855 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.052839    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 19:00:34.052855 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.052853    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 19:00:34.052962 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.052869    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 19:00:34.052962 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.052881    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:00:34.053680 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.053659    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:00:34.053756 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.053699    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:00:34.053756 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.053715    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:00:34.055018 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.055003    2579 policy_none.go:49] "None policy: Start"
Jul 29 19:00:34.055078 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.055024    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 19:00:34.055078 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.055038    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 19:00:34.057251 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.057237    2579 policy_none.go:47] "Start"
Jul 29 19:00:34.093035 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.092783    2579 manager.go:337] "Starting Device Plugin manager"
Jul 29 19:00:34.093155 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.093044    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 19:00:34.093155 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.093057    2579 server.go:87] "Starting device plugin registration server"
Jul 29 19:00:34.093350 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.093335    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 19:00:34.093392 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.093350    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 19:00:34.093468 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.093449    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 19:00:34.093560 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.093542    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 19:00:34.093560 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.093555    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 19:00:34.094569 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.094549    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 19:00:34.094674 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.094607    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-248.ec2.internal\" not found"
Jul 29 19:00:34.187945 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.187897    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 19:00:34.190206 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.189366    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 19:00:34.190206 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.189407    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 19:00:34.190206 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.189435    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 19:00:34.190206 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.189474    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 19:00:34.191759 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.191729    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:00:34.193627 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.193611    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:00:34.194491 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.194441    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:00:34.194491 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.194477    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:00:34.194491 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.194492    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:00:34.194633 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.194522    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.201560 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.201540    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.201684 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.201572    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-248.ec2.internal\": node \"ip-10-0-135-248.ec2.internal\" not found"
Jul 29 19:00:34.214174 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.214146    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found"
Jul 29 19:00:34.290510 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.290466    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal"]
Jul 29 19:00:34.290696 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.290607    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:00:34.291818 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.291794    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:00:34.291942 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.291829    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:00:34.291942 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.291841    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:00:34.294353 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.294338    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:00:34.294492 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.294476    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.294559 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.294515    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:00:34.295165 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.295139    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:00:34.295306 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.295177    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:00:34.295306 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.295187    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:00:34.295306 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.295210    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:00:34.295306 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.295233    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:00:34.295451 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.295310    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:00:34.297682 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.297667    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.297754 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.297693    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:00:34.298419 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.298399    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:00:34.298541 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.298429    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:00:34.298541 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.298441    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:00:34.314712 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.314675    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found"
Jul 29 19:00:34.319877 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.319850    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-248.ec2.internal\" not found" node="ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.323686 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.323668    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-248.ec2.internal\" not found" node="ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.333876 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.333838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ad2b80255b384c7f34ba4b0599ef380-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal\" (UID: \"8ad2b80255b384c7f34ba4b0599ef380\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.334011 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.333884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ad2b80255b384c7f34ba4b0599ef380-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal\" (UID: \"8ad2b80255b384c7f34ba4b0599ef380\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.334011 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.333907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c90e4ebfb2b0c26370b1fc329ec87357-config\") pod \"kube-apiserver-proxy-ip-10-0-135-248.ec2.internal\" (UID: \"c90e4ebfb2b0c26370b1fc329ec87357\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.415646 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.415608    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found"
Jul 29 19:00:34.435120 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.435039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ad2b80255b384c7f34ba4b0599ef380-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal\" (UID: \"8ad2b80255b384c7f34ba4b0599ef380\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.435242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.435146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c90e4ebfb2b0c26370b1fc329ec87357-config\") pod \"kube-apiserver-proxy-ip-10-0-135-248.ec2.internal\" (UID: \"c90e4ebfb2b0c26370b1fc329ec87357\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.435242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.435168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ad2b80255b384c7f34ba4b0599ef380-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal\" (UID: \"8ad2b80255b384c7f34ba4b0599ef380\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.435242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.435097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ad2b80255b384c7f34ba4b0599ef380-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal\" (UID: \"8ad2b80255b384c7f34ba4b0599ef380\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.435383 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.435244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c90e4ebfb2b0c26370b1fc329ec87357-config\") pod \"kube-apiserver-proxy-ip-10-0-135-248.ec2.internal\" (UID: \"c90e4ebfb2b0c26370b1fc329ec87357\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.435383 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.435248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ad2b80255b384c7f34ba4b0599ef380-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal\" (UID: \"8ad2b80255b384c7f34ba4b0599ef380\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.516592 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.516488    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found"
Jul 29 19:00:34.617116 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.617064    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found"
Jul 29 19:00:34.622272 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.622247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.626209 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.626175    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.717869 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.717828    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found"
Jul 29 19:00:34.822285 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:34.822164    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found"
Jul 29 19:00:34.860056 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.860030    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:00:34.886038 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.886011    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:00:34.931907 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.931880    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.943651 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.943630    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 19:00:34.944728 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.944713    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal"
Jul 29 19:00:34.950659 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.950641    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 19:00:34.967016 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.966983    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 19:00:34.967185 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.967159    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 19:00:34.967246 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.967219    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 19:00:34.967283 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:34.967226    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 19:00:35.006889 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.006842    2579 apiserver.go:52] "Watching apiserver"
Jul 29 19:00:35.027621 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.027594    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 19:00:35.028764 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.028737    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-zm2md","openshift-network-diagnostics/network-check-target-gsbwv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb","openshift-image-registry/node-ca-7tb67","openshift-multus/multus-additional-cni-plugins-zm58d","openshift-multus/multus-qpbzz","openshift-network-operator/iptables-alerter-mf8x6","openshift-ovn-kubernetes/ovnkube-node-pglrz","kube-system/global-pull-secret-syncer-gv68b","kube-system/konnectivity-agent-jmvb4","kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal","openshift-cluster-node-tuning-operator/tuned-sr9hs","openshift-dns/node-resolver-l9btm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal"]
Jul 29 19:00:35.029341 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.029305    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 18:55:34 +0000 UTC" deadline="2028-03-28 16:29:11.234604343 +0000 UTC"
Jul 29 19:00:35.029341 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.029332    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14589h28m36.205275437s"
Jul 29 19:00:35.031173 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.031157    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 19:00:35.031663 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.031633    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mf8x6"
Jul 29 19:00:35.033970 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.033953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l9btm"
Jul 29 19:00:35.035015 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.034846    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 19:00:35.035015 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.034953    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 19:00:35.037243 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.035008    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-94z5l\""
Jul 29 19:00:35.037243 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.035069    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:00:35.037623 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.037599    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.037755 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.037729    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 19:00:35.038405 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.038356    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vzf4x\""
Jul 29 19:00:35.038503 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.038446    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 19:00:35.039730 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.039711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab1c53d-2ed6-48b8-86b6-6244542b8750-tmp-dir\") pod \"node-resolver-l9btm\" (UID: \"fab1c53d-2ed6-48b8-86b6-6244542b8750\") " pod="openshift-dns/node-resolver-l9btm"
Jul 29 19:00:35.039822 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.039750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnh7n\" (UniqueName: \"kubernetes.io/projected/309ebf49-c66a-478e-af58-316c3d195af4-kube-api-access-fnh7n\") pod \"iptables-alerter-mf8x6\" (UID: \"309ebf49-c66a-478e-af58-316c3d195af4\") " pod="openshift-network-operator/iptables-alerter-mf8x6"
Jul 29 19:00:35.039822 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.039787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fab1c53d-2ed6-48b8-86b6-6244542b8750-hosts-file\") pod \"node-resolver-l9btm\" (UID: \"fab1c53d-2ed6-48b8-86b6-6244542b8750\") " pod="openshift-dns/node-resolver-l9btm"
Jul 29 19:00:35.039822 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.039817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9288c\" (UniqueName: \"kubernetes.io/projected/fab1c53d-2ed6-48b8-86b6-6244542b8750-kube-api-access-9288c\") pod \"node-resolver-l9btm\" (UID: \"fab1c53d-2ed6-48b8-86b6-6244542b8750\") " pod="openshift-dns/node-resolver-l9btm"
Jul 29 19:00:35.039966 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.039906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/309ebf49-c66a-478e-af58-316c3d195af4-iptables-alerter-script\") pod \"iptables-alerter-mf8x6\" (UID: \"309ebf49-c66a-478e-af58-316c3d195af4\") " pod="openshift-network-operator/iptables-alerter-mf8x6"
Jul 29 19:00:35.039966 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.039937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/309ebf49-c66a-478e-af58-316c3d195af4-host-slash\") pod \"iptables-alerter-mf8x6\" (UID: \"309ebf49-c66a-478e-af58-316c3d195af4\") " pod="openshift-network-operator/iptables-alerter-mf8x6"
Jul 29 19:00:35.040725 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.040708    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7tb67"
Jul 29 19:00:35.040982 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.040964    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 19:00:35.041167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.041149    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 19:00:35.041376 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.041363    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 19:00:35.041514 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.041500    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dkkdc\""
Jul 29 19:00:35.042750 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.042732    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 19:00:35.043115 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.043099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.043198 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.043151    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 19:00:35.043528 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.043504    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 19:00:35.043617 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.043558    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 19:00:35.043762 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.043746    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jmbcz\""
Jul 29 19:00:35.045528 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.045511    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.045724 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.045706    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mnmnv\""
Jul 29 19:00:35.045796 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.045740    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 19:00:35.046038 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.046025    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 19:00:35.046140 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.046123    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 19:00:35.046181 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.046132    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 19:00:35.046246 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.046233    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 19:00:35.047814 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.047794    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:35.047980 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.047955    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:00:35.047980 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.047973    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 19:00:35.048505 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.048486    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-66c5q\""
Jul 29 19:00:35.050135 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.050117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:35.050248 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.050185    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:00:35.052608 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.052590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:35.052695 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.052646    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:00:35.054962 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.054943    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jmvb4"
Jul 29 19:00:35.057508 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.057489    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.058292 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.058262    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2n7lp\""
Jul 29 19:00:35.058940 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.058924    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 19:00:35.059000 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.058924    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 19:00:35.060020 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.060006    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 19:00:35.060173 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.060160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.060452 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.060437    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7kp4n\""
Jul 29 19:00:35.061638 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.061608    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 19:00:35.061743 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.061689    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 19:00:35.061806 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.061742    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 19:00:35.061806 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.061762    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 19:00:35.061907 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.061768    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 19:00:35.062975 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.062955    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:00:35.063035 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.063023    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tv2tn\""
Jul 29 19:00:35.063084 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.062959    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 19:00:35.070216 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.070197    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t9zzr"
Jul 29 19:00:35.077700 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.077645    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t9zzr"
Jul 29 19:00:35.133203 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.133159    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 19:00:35.140500 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-env-overrides\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.140630 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-ovn-node-metrics-cert\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.140630 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140529    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/678bccec-6cea-4a28-9f6f-390c8ffec2c1-tmp\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.140630 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1acea82a-ad7e-4dfa-8737-c397979480fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.140630 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1acea82a-ad7e-4dfa-8737-c397979480fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.140630 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvfsb\" (UniqueName: \"kubernetes.io/projected/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-kube-api-access-dvfsb\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.140789 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-var-lib-cni-multus\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.140789 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-etc-kubernetes\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.140789 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fab1c53d-2ed6-48b8-86b6-6244542b8750-hosts-file\") pod \"node-resolver-l9btm\" (UID: \"fab1c53d-2ed6-48b8-86b6-6244542b8750\") " pod="openshift-dns/node-resolver-l9btm"
Jul 29 19:00:35.140789 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9288c\" (UniqueName: \"kubernetes.io/projected/fab1c53d-2ed6-48b8-86b6-6244542b8750-kube-api-access-9288c\") pod \"node-resolver-l9btm\" (UID: \"fab1c53d-2ed6-48b8-86b6-6244542b8750\") " pod="openshift-dns/node-resolver-l9btm"
Jul 29 19:00:35.140908 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-cni-binary-copy\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.140908 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140844    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fab1c53d-2ed6-48b8-86b6-6244542b8750-hosts-file\") pod \"node-resolver-l9btm\" (UID: \"fab1c53d-2ed6-48b8-86b6-6244542b8750\") " pod="openshift-dns/node-resolver-l9btm"
Jul 29 19:00:35.140908 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-run-netns\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.140908 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-cni-netd\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.140908 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-run-multus-certs\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.141053 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-var-lib-openvswitch\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.141053 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6b657d2b-5b8b-401b-ba94-b2c15ca3720f-agent-certs\") pod \"konnectivity-agent-jmvb4\" (UID: \"6b657d2b-5b8b-401b-ba94-b2c15ca3720f\") " pod="kube-system/konnectivity-agent-jmvb4"
Jul 29 19:00:35.141053 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-lib-modules\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.141053 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.140982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-hostroot\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.141053 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8n9\" (UniqueName: \"kubernetes.io/projected/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-kube-api-access-tz8n9\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.141053 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-node-log\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.141242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-ovnkube-config\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.141242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-registration-dir\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.141242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141129    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-modprobe-d\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.141242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-sysctl-d\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.141242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-multus-cni-dir\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.141242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-sys-fs\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.141242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-sysconfig\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.141469 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-kubernetes\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.141469 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltrhv\" (UniqueName: \"kubernetes.io/projected/841e2b86-5a6c-43d7-9919-da697a9ce802-kube-api-access-ltrhv\") pod \"node-ca-7tb67\" (UID: \"841e2b86-5a6c-43d7-9919-da697a9ce802\") " pod="openshift-image-registry/node-ca-7tb67"
Jul 29 19:00:35.141469 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1acea82a-ad7e-4dfa-8737-c397979480fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.141469 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-multus-conf-dir\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.141469 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-systemd-units\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.141469 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-run-ovn\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.141469 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxb2m\" (UniqueName: \"kubernetes.io/projected/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-kube-api-access-kxb2m\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.141469 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:35.141469 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-systemd\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.141469 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-os-release\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.141749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-run-ovn-kubernetes\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.141749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-run\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.141749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/841e2b86-5a6c-43d7-9919-da697a9ce802-host\") pod \"node-ca-7tb67\" (UID: \"841e2b86-5a6c-43d7-9919-da697a9ce802\") " pod="openshift-image-registry/node-ca-7tb67"
Jul 29 19:00:35.141749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-etc-selinux\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.141749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-etc-openvswitch\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.141749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-log-socket\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.141749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-kubelet-dir\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.141749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4drdp\" (UniqueName: \"kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp\") pod \"network-check-target-gsbwv\" (UID: \"07f5037f-a52e-4722-898b-8e3c53085d4b\") " pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:35.141749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-multus-socket-dir-parent\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.141749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-run-k8s-cni-cncf-io\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.141749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-var-lib-cni-bin\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.141749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-run-systemd\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.141749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-device-dir\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.141749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1acea82a-ad7e-4dfa-8737-c397979480fb-cnibin\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1acea82a-ad7e-4dfa-8737-c397979480fb-os-release\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-cnibin\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-multus-daemon-config\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-ovnkube-script-lib\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-socket-dir\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141872    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-var-lib-kubelet\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-host\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/898bc211-2d04-4fe7-b981-4ce7f9c0a856-dbus\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxp4\" (UniqueName: \"kubernetes.io/projected/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-kube-api-access-7jxp4\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.141992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmz8z\" (UniqueName: \"kubernetes.io/projected/678bccec-6cea-4a28-9f6f-390c8ffec2c1-kube-api-access-gmz8z\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1acea82a-ad7e-4dfa-8737-c397979480fb-system-cni-dir\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj2mv\" (UniqueName: \"kubernetes.io/projected/1acea82a-ad7e-4dfa-8737-c397979480fb-kube-api-access-sj2mv\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab1c53d-2ed6-48b8-86b6-6244542b8750-tmp-dir\") pod \"node-resolver-l9btm\" (UID: \"fab1c53d-2ed6-48b8-86b6-6244542b8750\") " pod="openshift-dns/node-resolver-l9btm"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-kubelet\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.142154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-slash\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6b657d2b-5b8b-401b-ba94-b2c15ca3720f-konnectivity-ca\") pod \"konnectivity-agent-jmvb4\" (UID: \"6b657d2b-5b8b-401b-ba94-b2c15ca3720f\") " pod="kube-system/konnectivity-agent-jmvb4"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-sysctl-conf\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-sys\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-tuned\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-var-lib-kubelet\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/841e2b86-5a6c-43d7-9919-da697a9ce802-serviceca\") pod \"node-ca-7tb67\" (UID: \"841e2b86-5a6c-43d7-9919-da697a9ce802\") " pod="openshift-image-registry/node-ca-7tb67"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/309ebf49-c66a-478e-af58-316c3d195af4-host-slash\") pod \"iptables-alerter-mf8x6\" (UID: \"309ebf49-c66a-478e-af58-316c3d195af4\") " pod="openshift-network-operator/iptables-alerter-mf8x6"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-run-netns\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-cni-bin\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/309ebf49-c66a-478e-af58-316c3d195af4-host-slash\") pod \"iptables-alerter-mf8x6\" (UID: \"309ebf49-c66a-478e-af58-316c3d195af4\") " pod="openshift-network-operator/iptables-alerter-mf8x6"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/898bc211-2d04-4fe7-b981-4ce7f9c0a856-kubelet-config\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnh7n\" (UniqueName: \"kubernetes.io/projected/309ebf49-c66a-478e-af58-316c3d195af4-kube-api-access-fnh7n\") pod \"iptables-alerter-mf8x6\" (UID: \"309ebf49-c66a-478e-af58-316c3d195af4\") " pod="openshift-network-operator/iptables-alerter-mf8x6"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1acea82a-ad7e-4dfa-8737-c397979480fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/309ebf49-c66a-478e-af58-316c3d195af4-iptables-alerter-script\") pod \"iptables-alerter-mf8x6\" (UID: \"309ebf49-c66a-478e-af58-316c3d195af4\") " pod="openshift-network-operator/iptables-alerter-mf8x6"
Jul 29 19:00:35.142749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-system-cni-dir\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.143218 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.142545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-run-openvswitch\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.143218 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.143083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab1c53d-2ed6-48b8-86b6-6244542b8750-tmp-dir\") pod \"node-resolver-l9btm\" (UID: \"fab1c53d-2ed6-48b8-86b6-6244542b8750\") " pod="openshift-dns/node-resolver-l9btm"
Jul 29 19:00:35.143656 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.143638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/309ebf49-c66a-478e-af58-316c3d195af4-iptables-alerter-script\") pod \"iptables-alerter-mf8x6\" (UID: \"309ebf49-c66a-478e-af58-316c3d195af4\") " pod="openshift-network-operator/iptables-alerter-mf8x6"
Jul 29 19:00:35.149355 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.149327    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 19:00:35.152888 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.152863    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnh7n\" (UniqueName: \"kubernetes.io/projected/309ebf49-c66a-478e-af58-316c3d195af4-kube-api-access-fnh7n\") pod \"iptables-alerter-mf8x6\" (UID: \"309ebf49-c66a-478e-af58-316c3d195af4\") " pod="openshift-network-operator/iptables-alerter-mf8x6"
Jul 29 19:00:35.153010 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.152920    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9288c\" (UniqueName: \"kubernetes.io/projected/fab1c53d-2ed6-48b8-86b6-6244542b8750-kube-api-access-9288c\") pod \"node-resolver-l9btm\" (UID: \"fab1c53d-2ed6-48b8-86b6-6244542b8750\") " pod="openshift-dns/node-resolver-l9btm"
Jul 29 19:00:35.243469 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxp4\" (UniqueName: \"kubernetes.io/projected/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-kube-api-access-7jxp4\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:35.243469 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmz8z\" (UniqueName: \"kubernetes.io/projected/678bccec-6cea-4a28-9f6f-390c8ffec2c1-kube-api-access-gmz8z\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.243776 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1acea82a-ad7e-4dfa-8737-c397979480fb-system-cni-dir\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.243776 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj2mv\" (UniqueName: \"kubernetes.io/projected/1acea82a-ad7e-4dfa-8737-c397979480fb-kube-api-access-sj2mv\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.243776 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-kubelet\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.243776 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1acea82a-ad7e-4dfa-8737-c397979480fb-system-cni-dir\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.243776 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-kubelet\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.243776 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-slash\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.243776 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:35.243776 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6b657d2b-5b8b-401b-ba94-b2c15ca3720f-konnectivity-ca\") pod \"konnectivity-agent-jmvb4\" (UID: \"6b657d2b-5b8b-401b-ba94-b2c15ca3720f\") " pod="kube-system/konnectivity-agent-jmvb4"
Jul 29 19:00:35.243776 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-sysctl-conf\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.243776 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-slash\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.243793    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243864    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-sysctl-conf\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-sys\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.243905    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs podName:1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:35.743863048 +0000 UTC m=+2.043535551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs") pod "network-metrics-daemon-zm2md" (UID: "1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-sys\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-tuned\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.243984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-var-lib-kubelet\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/841e2b86-5a6c-43d7-9919-da697a9ce802-serviceca\") pod \"node-ca-7tb67\" (UID: \"841e2b86-5a6c-43d7-9919-da697a9ce802\") " pod="openshift-image-registry/node-ca-7tb67"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-run-netns\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-var-lib-kubelet\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-cni-bin\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/898bc211-2d04-4fe7-b981-4ce7f9c0a856-kubelet-config\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-run-netns\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1acea82a-ad7e-4dfa-8737-c397979480fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-cni-bin\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-system-cni-dir\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.244360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/898bc211-2d04-4fe7-b981-4ce7f9c0a856-kubelet-config\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-run-openvswitch\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-env-overrides\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244274    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-system-cni-dir\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-ovn-node-metrics-cert\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/678bccec-6cea-4a28-9f6f-390c8ffec2c1-tmp\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-run-openvswitch\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1acea82a-ad7e-4dfa-8737-c397979480fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6b657d2b-5b8b-401b-ba94-b2c15ca3720f-konnectivity-ca\") pod \"konnectivity-agent-jmvb4\" (UID: \"6b657d2b-5b8b-401b-ba94-b2c15ca3720f\") " pod="kube-system/konnectivity-agent-jmvb4"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1acea82a-ad7e-4dfa-8737-c397979480fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvfsb\" (UniqueName: \"kubernetes.io/projected/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-kube-api-access-dvfsb\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244570    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/841e2b86-5a6c-43d7-9919-da697a9ce802-serviceca\") pod \"node-ca-7tb67\" (UID: \"841e2b86-5a6c-43d7-9919-da697a9ce802\") " pod="openshift-image-registry/node-ca-7tb67"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-var-lib-cni-multus\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-etc-kubernetes\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-var-lib-cni-multus\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-cni-binary-copy\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-run-netns\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.245167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-cni-netd\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-run-multus-certs\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-var-lib-openvswitch\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6b657d2b-5b8b-401b-ba94-b2c15ca3720f-agent-certs\") pod \"konnectivity-agent-jmvb4\" (UID: \"6b657d2b-5b8b-401b-ba94-b2c15ca3720f\") " pod="kube-system/konnectivity-agent-jmvb4"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1acea82a-ad7e-4dfa-8737-c397979480fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-lib-modules\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-cni-netd\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-hostroot\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8n9\" (UniqueName: \"kubernetes.io/projected/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-kube-api-access-tz8n9\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-node-log\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244915    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-run-multus-certs\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-ovnkube-config\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-lib-modules\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-registration-dir\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-modprobe-d\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.244997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-run-netns\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-sysctl-d\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.246101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-multus-cni-dir\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-sys-fs\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-sysconfig\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245099    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-var-lib-openvswitch\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-hostroot\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-sysconfig\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-kubernetes\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1acea82a-ad7e-4dfa-8737-c397979480fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltrhv\" (UniqueName: \"kubernetes.io/projected/841e2b86-5a6c-43d7-9919-da697a9ce802-kube-api-access-ltrhv\") pod \"node-ca-7tb67\" (UID: \"841e2b86-5a6c-43d7-9919-da697a9ce802\") " pod="openshift-image-registry/node-ca-7tb67"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1acea82a-ad7e-4dfa-8737-c397979480fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-multus-conf-dir\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-systemd-units\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-run-ovn\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxb2m\" (UniqueName: \"kubernetes.io/projected/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-kube-api-access-kxb2m\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-node-log\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-systemd\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.246910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.246610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-os-release\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.246643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-run-ovn-kubernetes\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-systemd\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-etc-kubernetes\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-systemd-units\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-run-ovn\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.246720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-run\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.245853    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.246797    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret podName:898bc211-2d04-4fe7-b981-4ce7f9c0a856 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:35.746776121 +0000 UTC m=+2.046448602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret") pod "global-pull-secret-syncer-gv68b" (UID: "898bc211-2d04-4fe7-b981-4ce7f9c0a856") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-multus-conf-dir\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.246795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-run-ovn-kubernetes\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-modprobe-d\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.246043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-multus-cni-dir\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.246038    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-registration-dir\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-cni-binary-copy\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.246103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-env-overrides\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.246170    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-sysctl-d\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.246667    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-run\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.247766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/841e2b86-5a6c-43d7-9919-da697a9ce802-host\") pod \"node-ca-7tb67\" (UID: \"841e2b86-5a6c-43d7-9919-da697a9ce802\") " pod="openshift-image-registry/node-ca-7tb67"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-etc-selinux\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.246087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-sys-fs\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-etc-openvswitch\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-log-socket\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-kubelet-dir\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247364    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/841e2b86-5a6c-43d7-9919-da697a9ce802-host\") pod \"node-ca-7tb67\" (UID: \"841e2b86-5a6c-43d7-9919-da697a9ce802\") " pod="openshift-image-registry/node-ca-7tb67"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.246516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-ovnkube-config\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drdp\" (UniqueName: \"kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp\") pod \"network-check-target-gsbwv\" (UID: \"07f5037f-a52e-4722-898b-8e3c53085d4b\") " pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1acea82a-ad7e-4dfa-8737-c397979480fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-kubernetes\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-os-release\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-kubelet-dir\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/678bccec-6cea-4a28-9f6f-390c8ffec2c1-etc-tuned\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/678bccec-6cea-4a28-9f6f-390c8ffec2c1-tmp\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-etc-selinux\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247395    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-etc-openvswitch\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.248569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-multus-socket-dir-parent\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-log-socket\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-run-k8s-cni-cncf-io\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-var-lib-cni-bin\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-multus-socket-dir-parent\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-run-systemd\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-run-k8s-cni-cncf-io\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247669    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-run-systemd\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.245934    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1acea82a-ad7e-4dfa-8737-c397979480fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-host-var-lib-cni-bin\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-device-dir\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1acea82a-ad7e-4dfa-8737-c397979480fb-cnibin\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-device-dir\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1acea82a-ad7e-4dfa-8737-c397979480fb-os-release\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-cnibin\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-multus-daemon-config\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1acea82a-ad7e-4dfa-8737-c397979480fb-cnibin\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.249223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-ovnkube-script-lib\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247904    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1acea82a-ad7e-4dfa-8737-c397979480fb-os-release\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-socket-dir\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-var-lib-kubelet\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-host\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.247926    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-cnibin\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.248033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.248065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-var-lib-kubelet\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.248071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/898bc211-2d04-4fe7-b981-4ce7f9c0a856-dbus\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.248137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/678bccec-6cea-4a28-9f6f-390c8ffec2c1-host\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.248168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/898bc211-2d04-4fe7-b981-4ce7f9c0a856-dbus\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.248218    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.248351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-socket-dir\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.248450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-multus-daemon-config\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.248491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-ovnkube-script-lib\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.248689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6b657d2b-5b8b-401b-ba94-b2c15ca3720f-agent-certs\") pod \"konnectivity-agent-jmvb4\" (UID: \"6b657d2b-5b8b-401b-ba94-b2c15ca3720f\") " pod="kube-system/konnectivity-agent-jmvb4"
Jul 29 19:00:35.249760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.248802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-ovn-node-metrics-cert\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.255357 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.255330    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:00:35.255357 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.255358    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:00:35.255577 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.255374    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4drdp for pod openshift-network-diagnostics/network-check-target-gsbwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:00:35.256072 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.255713    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp podName:07f5037f-a52e-4722-898b-8e3c53085d4b nodeName:}" failed. No retries permitted until 2026-07-29 19:00:35.755690002 +0000 UTC m=+2.055362501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4drdp" (UniqueName: "kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp") pod "network-check-target-gsbwv" (UID: "07f5037f-a52e-4722-898b-8e3c53085d4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:00:35.258057 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.257367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxp4\" (UniqueName: \"kubernetes.io/projected/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-kube-api-access-7jxp4\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:35.258057 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.257910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj2mv\" (UniqueName: \"kubernetes.io/projected/1acea82a-ad7e-4dfa-8737-c397979480fb-kube-api-access-sj2mv\") pod \"multus-additional-cni-plugins-zm58d\" (UID: \"1acea82a-ad7e-4dfa-8737-c397979480fb\") " pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.258057 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.257994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvfsb\" (UniqueName: \"kubernetes.io/projected/e6d8a6bf-13b0-4efb-ab11-1472ef36dd09-kube-api-access-dvfsb\") pod \"aws-ebs-csi-driver-node-94zmb\" (UID: \"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.258355 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.258196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltrhv\" (UniqueName: \"kubernetes.io/projected/841e2b86-5a6c-43d7-9919-da697a9ce802-kube-api-access-ltrhv\") pod \"node-ca-7tb67\" (UID: \"841e2b86-5a6c-43d7-9919-da697a9ce802\") " pod="openshift-image-registry/node-ca-7tb67"
Jul 29 19:00:35.258502 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.258482    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmz8z\" (UniqueName: \"kubernetes.io/projected/678bccec-6cea-4a28-9f6f-390c8ffec2c1-kube-api-access-gmz8z\") pod \"tuned-sr9hs\" (UID: \"678bccec-6cea-4a28-9f6f-390c8ffec2c1\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.258502 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.258495    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8n9\" (UniqueName: \"kubernetes.io/projected/7e80e4a8-0ff2-4412-a9d7-826cfce55e8c-kube-api-access-tz8n9\") pod \"multus-qpbzz\" (UID: \"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c\") " pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.258685 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.258668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxb2m\" (UniqueName: \"kubernetes.io/projected/d0cc9ff8-42b8-4517-904b-c6a12d9fb37f-kube-api-access-kxb2m\") pod \"ovnkube-node-pglrz\" (UID: \"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.368063 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.367942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mf8x6"
Jul 29 19:00:35.383987 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.383954    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l9btm"
Jul 29 19:00:35.400568 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.400531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb"
Jul 29 19:00:35.419894 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.419863    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7tb67"
Jul 29 19:00:35.437547 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.437518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zm58d"
Jul 29 19:00:35.458589 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.458551    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qpbzz"
Jul 29 19:00:35.477085 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.477050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jmvb4"
Jul 29 19:00:35.483391 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.483363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:00:35.487442 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.487424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sr9hs"
Jul 29 19:00:35.579705 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.579672    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:00:35.732289 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:35.732219    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad2b80255b384c7f34ba4b0599ef380.slice/crio-80e0b5bbeb4f27e7b91fbf4f98cb9c12c05e42132af735e0d029a563a6afe444 WatchSource:0}: Error finding container 80e0b5bbeb4f27e7b91fbf4f98cb9c12c05e42132af735e0d029a563a6afe444: Status 404 returned error can't find the container with id 80e0b5bbeb4f27e7b91fbf4f98cb9c12c05e42132af735e0d029a563a6afe444
Jul 29 19:00:35.737818 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.737798    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:00:35.750726 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.750694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:35.750882 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.750751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:35.750882 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.750853    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:00:35.750882 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.750856    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:00:35.751013 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.750912    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs podName:1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:36.750895972 +0000 UTC m=+3.050568452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs") pod "network-metrics-daemon-zm2md" (UID: "1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:00:35.751013 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.750926    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret podName:898bc211-2d04-4fe7-b981-4ce7f9c0a856 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:36.750919479 +0000 UTC m=+3.050591959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret") pod "global-pull-secret-syncer-gv68b" (UID: "898bc211-2d04-4fe7-b981-4ce7f9c0a856") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:00:35.780600 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:35.780562    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309ebf49_c66a_478e_af58_316c3d195af4.slice/crio-db648a28ab89d761472ca0d948594c5e3ac22a324b5b652d12b90ba1c148ed2c WatchSource:0}: Error finding container db648a28ab89d761472ca0d948594c5e3ac22a324b5b652d12b90ba1c148ed2c: Status 404 returned error can't find the container with id db648a28ab89d761472ca0d948594c5e3ac22a324b5b652d12b90ba1c148ed2c
Jul 29 19:00:35.851942 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.851902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drdp\" (UniqueName: \"kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp\") pod \"network-check-target-gsbwv\" (UID: \"07f5037f-a52e-4722-898b-8e3c53085d4b\") " pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:35.852162 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.852046    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:00:35.852162 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.852063    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:00:35.852162 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.852073    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4drdp for pod openshift-network-diagnostics/network-check-target-gsbwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:00:35.852162 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:35.852130    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp podName:07f5037f-a52e-4722-898b-8e3c53085d4b nodeName:}" failed. No retries permitted until 2026-07-29 19:00:36.852114611 +0000 UTC m=+3.151787096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4drdp" (UniqueName: "kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp") pod "network-check-target-gsbwv" (UID: "07f5037f-a52e-4722-898b-8e3c53085d4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:00:35.856669 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:35.856636    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab1c53d_2ed6_48b8_86b6_6244542b8750.slice/crio-014e3917e39f89f793c316906cbb954d262757f594d947131842d4cc2cd30549 WatchSource:0}: Error finding container 014e3917e39f89f793c316906cbb954d262757f594d947131842d4cc2cd30549: Status 404 returned error can't find the container with id 014e3917e39f89f793c316906cbb954d262757f594d947131842d4cc2cd30549
Jul 29 19:00:35.887672 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:35.887643    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:00:35.913685 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:35.913647    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0cc9ff8_42b8_4517_904b_c6a12d9fb37f.slice/crio-ecd11b01397dbaea52f61ebd197dd3870038f17f1fed25fcfc67e6d8ca9a5697 WatchSource:0}: Error finding container ecd11b01397dbaea52f61ebd197dd3870038f17f1fed25fcfc67e6d8ca9a5697: Status 404 returned error can't find the container with id ecd11b01397dbaea52f61ebd197dd3870038f17f1fed25fcfc67e6d8ca9a5697
Jul 29 19:00:35.917474 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:35.917439    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841e2b86_5a6c_43d7_9919_da697a9ce802.slice/crio-78a0e5c0ca423734823ad75a37d599d499fd8ecd9ddc28ef84fd21763e6e9147 WatchSource:0}: Error finding container 78a0e5c0ca423734823ad75a37d599d499fd8ecd9ddc28ef84fd21763e6e9147: Status 404 returned error can't find the container with id 78a0e5c0ca423734823ad75a37d599d499fd8ecd9ddc28ef84fd21763e6e9147
Jul 29 19:00:35.920222 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:35.919997    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d8a6bf_13b0_4efb_ab11_1472ef36dd09.slice/crio-2274c36d2b93203aa6f0012819f031a9459358777ff2f2321bc2e8dfbb7f5421 WatchSource:0}: Error finding container 2274c36d2b93203aa6f0012819f031a9459358777ff2f2321bc2e8dfbb7f5421: Status 404 returned error can't find the container with id 2274c36d2b93203aa6f0012819f031a9459358777ff2f2321bc2e8dfbb7f5421
Jul 29 19:00:35.945111 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:35.945075    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b657d2b_5b8b_401b_ba94_b2c15ca3720f.slice/crio-88b73222998a9038e2b7a02f7b31afa210cd0056ddecdf6f42032997a1b68d8f WatchSource:0}: Error finding container 88b73222998a9038e2b7a02f7b31afa210cd0056ddecdf6f42032997a1b68d8f: Status 404 returned error can't find the container with id 88b73222998a9038e2b7a02f7b31afa210cd0056ddecdf6f42032997a1b68d8f
Jul 29 19:00:36.011689 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:36.011647    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acea82a_ad7e_4dfa_8737_c397979480fb.slice/crio-8b69ca69c254796d09ddb77037aa63528308d72c851df767e00feee0f35c37f8 WatchSource:0}: Error finding container 8b69ca69c254796d09ddb77037aa63528308d72c851df767e00feee0f35c37f8: Status 404 returned error can't find the container with id 8b69ca69c254796d09ddb77037aa63528308d72c851df767e00feee0f35c37f8
Jul 29 19:00:36.080944 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.080869    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 18:55:35 +0000 UTC" deadline="2028-05-14 18:48:19.349250888 +0000 UTC"
Jul 29 19:00:36.080944 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.080937    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15719h47m43.2683175s"
Jul 29 19:00:36.109523 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:36.109488    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e80e4a8_0ff2_4412_a9d7_826cfce55e8c.slice/crio-8198d7a4a0d4f5985279b77ce1eb8baae62adacace680715c7932240be2464ac WatchSource:0}: Error finding container 8198d7a4a0d4f5985279b77ce1eb8baae62adacace680715c7932240be2464ac: Status 404 returned error can't find the container with id 8198d7a4a0d4f5985279b77ce1eb8baae62adacace680715c7932240be2464ac
Jul 29 19:00:36.195441 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.195368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zm58d" event={"ID":"1acea82a-ad7e-4dfa-8737-c397979480fb","Type":"ContainerStarted","Data":"8b69ca69c254796d09ddb77037aa63528308d72c851df767e00feee0f35c37f8"}
Jul 29 19:00:36.200584 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.200528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jmvb4" event={"ID":"6b657d2b-5b8b-401b-ba94-b2c15ca3720f","Type":"ContainerStarted","Data":"88b73222998a9038e2b7a02f7b31afa210cd0056ddecdf6f42032997a1b68d8f"}
Jul 29 19:00:36.202064 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.202013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb" event={"ID":"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09","Type":"ContainerStarted","Data":"2274c36d2b93203aa6f0012819f031a9459358777ff2f2321bc2e8dfbb7f5421"}
Jul 29 19:00:36.203289 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.203254    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qpbzz" event={"ID":"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c","Type":"ContainerStarted","Data":"8198d7a4a0d4f5985279b77ce1eb8baae62adacace680715c7932240be2464ac"}
Jul 29 19:00:36.204672 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.204642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7tb67" event={"ID":"841e2b86-5a6c-43d7-9919-da697a9ce802","Type":"ContainerStarted","Data":"78a0e5c0ca423734823ad75a37d599d499fd8ecd9ddc28ef84fd21763e6e9147"}
Jul 29 19:00:36.205988 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.205954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz" event={"ID":"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f","Type":"ContainerStarted","Data":"ecd11b01397dbaea52f61ebd197dd3870038f17f1fed25fcfc67e6d8ca9a5697"}
Jul 29 19:00:36.207330 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.207301    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l9btm" event={"ID":"fab1c53d-2ed6-48b8-86b6-6244542b8750","Type":"ContainerStarted","Data":"014e3917e39f89f793c316906cbb954d262757f594d947131842d4cc2cd30549"}
Jul 29 19:00:36.208547 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.208521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mf8x6" event={"ID":"309ebf49-c66a-478e-af58-316c3d195af4","Type":"ContainerStarted","Data":"db648a28ab89d761472ca0d948594c5e3ac22a324b5b652d12b90ba1c148ed2c"}
Jul 29 19:00:36.209773 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.209744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" event={"ID":"8ad2b80255b384c7f34ba4b0599ef380","Type":"ContainerStarted","Data":"80e0b5bbeb4f27e7b91fbf4f98cb9c12c05e42132af735e0d029a563a6afe444"}
Jul 29 19:00:36.498211 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:36.498146    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678bccec_6cea_4a28_9f6f_390c8ffec2c1.slice/crio-b0229242545f140e727734bcddaae8425ec4dc2996fe1e111b0e82ea9485df80 WatchSource:0}: Error finding container b0229242545f140e727734bcddaae8425ec4dc2996fe1e111b0e82ea9485df80: Status 404 returned error can't find the container with id b0229242545f140e727734bcddaae8425ec4dc2996fe1e111b0e82ea9485df80
Jul 29 19:00:36.759708 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.759598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:36.759708 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.759698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:36.760043 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:36.759858    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:00:36.760043 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:36.759946    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs podName:1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:38.759920264 +0000 UTC m=+5.059592766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs") pod "network-metrics-daemon-zm2md" (UID: "1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:00:36.760484 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:36.760442    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:00:36.760608 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:36.760529    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret podName:898bc211-2d04-4fe7-b981-4ce7f9c0a856 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:38.760507466 +0000 UTC m=+5.060179961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret") pod "global-pull-secret-syncer-gv68b" (UID: "898bc211-2d04-4fe7-b981-4ce7f9c0a856") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:00:36.860381 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.860329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drdp\" (UniqueName: \"kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp\") pod \"network-check-target-gsbwv\" (UID: \"07f5037f-a52e-4722-898b-8e3c53085d4b\") " pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:36.860604 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:36.860560    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:00:36.860604 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:36.860586    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:00:36.860604 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:36.860602    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4drdp for pod openshift-network-diagnostics/network-check-target-gsbwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:00:36.860784 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:36.860674    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp podName:07f5037f-a52e-4722-898b-8e3c53085d4b nodeName:}" failed. No retries permitted until 2026-07-29 19:00:38.860651245 +0000 UTC m=+5.160323730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4drdp" (UniqueName: "kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp") pod "network-check-target-gsbwv" (UID: "07f5037f-a52e-4722-898b-8e3c53085d4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:00:36.959992 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:36.959949    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:00:37.081735 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:37.081594    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 18:55:35 +0000 UTC" deadline="2028-03-03 23:39:21.084067006 +0000 UTC"
Jul 29 19:00:37.081735 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:37.081673    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13996h38m44.002398995s"
Jul 29 19:00:37.190628 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:37.190577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:37.190858 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:37.190596    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:37.190858 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:37.190738    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:00:37.190858 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:37.190763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:37.191054 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:37.190889    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:00:37.191054 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:37.190995    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:00:37.223125 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:37.223078    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sr9hs" event={"ID":"678bccec-6cea-4a28-9f6f-390c8ffec2c1","Type":"ContainerStarted","Data":"b0229242545f140e727734bcddaae8425ec4dc2996fe1e111b0e82ea9485df80"}
Jul 29 19:00:38.041604 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:38.041564    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:00:38.396874 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:00:38.396824    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90e4ebfb2b0c26370b1fc329ec87357.slice/crio-ac05a39ef7317eb2a87f2b607fbdf6fb52f78c53046a0cbbb9e00fb019e4f0a4 WatchSource:0}: Error finding container ac05a39ef7317eb2a87f2b607fbdf6fb52f78c53046a0cbbb9e00fb019e4f0a4: Status 404 returned error can't find the container with id ac05a39ef7317eb2a87f2b607fbdf6fb52f78c53046a0cbbb9e00fb019e4f0a4
Jul 29 19:00:38.778475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:38.778360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:38.778475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:38.778455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:38.778742 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:38.778598    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:00:38.778742 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:38.778673    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret podName:898bc211-2d04-4fe7-b981-4ce7f9c0a856 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:42.778650224 +0000 UTC m=+9.078322707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret") pod "global-pull-secret-syncer-gv68b" (UID: "898bc211-2d04-4fe7-b981-4ce7f9c0a856") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:00:38.779194 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:38.779170    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:00:38.779329 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:38.779246    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs podName:1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:42.779225581 +0000 UTC m=+9.078898086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs") pod "network-metrics-daemon-zm2md" (UID: "1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:00:38.879908 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:38.879853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drdp\" (UniqueName: \"kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp\") pod \"network-check-target-gsbwv\" (UID: \"07f5037f-a52e-4722-898b-8e3c53085d4b\") " pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:38.880145 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:38.880085    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:00:38.880145 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:38.880111    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:00:38.880145 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:38.880126    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4drdp for pod openshift-network-diagnostics/network-check-target-gsbwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:00:38.880367 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:38.880209    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp podName:07f5037f-a52e-4722-898b-8e3c53085d4b nodeName:}" failed. No retries permitted until 2026-07-29 19:00:42.880182469 +0000 UTC m=+9.179854974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4drdp" (UniqueName: "kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp") pod "network-check-target-gsbwv" (UID: "07f5037f-a52e-4722-898b-8e3c53085d4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:00:39.190756 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:39.190654    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:39.191261 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:39.190837    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:00:39.191261 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:39.190922    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:39.191261 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:39.191009    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:00:39.191261 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:39.191070    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:39.191261 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:39.191206    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:00:39.237906 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:39.237803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal" event={"ID":"c90e4ebfb2b0c26370b1fc329ec87357","Type":"ContainerStarted","Data":"ac05a39ef7317eb2a87f2b607fbdf6fb52f78c53046a0cbbb9e00fb019e4f0a4"}
Jul 29 19:00:41.190593 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:41.190361    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:41.191146 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:41.190703    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:00:41.191146 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:41.190748    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:41.191146 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:41.190822    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:00:41.191146 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:41.190877    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:41.191146 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:41.190956    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:00:41.247095 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:41.247047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" event={"ID":"8ad2b80255b384c7f34ba4b0599ef380","Type":"ContainerStarted","Data":"02b33ac4f3df49325e6735cd407dec926d41be2a97a5df9a8c75e7da978e17f6"}
Jul 29 19:00:42.252453 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:42.252376    2579 generic.go:356] "Generic (PLEG): container finished" podID="8ad2b80255b384c7f34ba4b0599ef380" containerID="02b33ac4f3df49325e6735cd407dec926d41be2a97a5df9a8c75e7da978e17f6" exitCode=0
Jul 29 19:00:42.252453 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:42.252426    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" event={"ID":"8ad2b80255b384c7f34ba4b0599ef380","Type":"ContainerDied","Data":"02b33ac4f3df49325e6735cd407dec926d41be2a97a5df9a8c75e7da978e17f6"}
Jul 29 19:00:42.822292 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:42.822235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:42.822519 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:42.822334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:42.822519 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:42.822512    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:00:42.822635 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:42.822585    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs podName:1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:50.822563667 +0000 UTC m=+17.122236160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs") pod "network-metrics-daemon-zm2md" (UID: "1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:00:42.822889 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:42.822866    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:00:42.822949 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:42.822942    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret podName:898bc211-2d04-4fe7-b981-4ce7f9c0a856 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:50.82292324 +0000 UTC m=+17.122595732 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret") pod "global-pull-secret-syncer-gv68b" (UID: "898bc211-2d04-4fe7-b981-4ce7f9c0a856") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:00:42.923767 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:42.923729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drdp\" (UniqueName: \"kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp\") pod \"network-check-target-gsbwv\" (UID: \"07f5037f-a52e-4722-898b-8e3c53085d4b\") " pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:42.923944 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:42.923923    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:00:42.923944 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:42.923944    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:00:42.924069 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:42.923953    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4drdp for pod openshift-network-diagnostics/network-check-target-gsbwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:00:42.924069 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:42.923999    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp podName:07f5037f-a52e-4722-898b-8e3c53085d4b nodeName:}" failed. No retries permitted until 2026-07-29 19:00:50.92398726 +0000 UTC m=+17.223659738 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4drdp" (UniqueName: "kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp") pod "network-check-target-gsbwv" (UID: "07f5037f-a52e-4722-898b-8e3c53085d4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:00:43.190862 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:43.190767    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:43.191034 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:43.190916    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:00:43.191837 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:43.191389    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:43.191837 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:43.191573    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:00:43.191837 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:43.191653    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:43.191837 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:43.191794    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:00:45.190644 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:45.190603    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:45.190644 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:45.190632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:45.191351 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:45.190630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:45.191351 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:45.190754    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:00:45.191351 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:45.190906    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:00:45.191351 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:45.191016    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:00:47.190465 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:47.190426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:47.190998 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:47.190436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:47.190998 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:47.190566    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:00:47.190998 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:47.190645    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:00:47.190998 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:47.190446    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:47.190998 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:47.190758    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:00:49.190719 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:49.190672    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:49.190719 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:49.190714    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:49.191262 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:49.190672    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:49.191262 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:49.190830    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:00:49.191262 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:49.190937    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:00:49.191262 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:49.191029    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:00:50.888614 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:50.888502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:50.888614 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:50.888593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:50.889131 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:50.888649    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:00:50.889131 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:50.888676    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:00:50.889131 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:50.888728    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret podName:898bc211-2d04-4fe7-b981-4ce7f9c0a856 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:06.888706935 +0000 UTC m=+33.188379429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret") pod "global-pull-secret-syncer-gv68b" (UID: "898bc211-2d04-4fe7-b981-4ce7f9c0a856") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:00:50.889131 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:50.888745    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs podName:1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:06.888738422 +0000 UTC m=+33.188410901 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs") pod "network-metrics-daemon-zm2md" (UID: "1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:00:50.989425 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:50.989367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drdp\" (UniqueName: \"kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp\") pod \"network-check-target-gsbwv\" (UID: \"07f5037f-a52e-4722-898b-8e3c53085d4b\") " pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:50.989662 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:50.989626    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:00:50.989662 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:50.989667    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:00:50.989887 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:50.989692    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4drdp for pod openshift-network-diagnostics/network-check-target-gsbwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:00:50.989887 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:50.989782    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp podName:07f5037f-a52e-4722-898b-8e3c53085d4b nodeName:}" failed. No retries permitted until 2026-07-29 19:01:06.989755074 +0000 UTC m=+33.289427583 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4drdp" (UniqueName: "kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp") pod "network-check-target-gsbwv" (UID: "07f5037f-a52e-4722-898b-8e3c53085d4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:00:51.190785 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:51.190687    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:51.190785 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:51.190703    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:51.191033 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:51.190834    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:00:51.191033 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:51.190717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:51.191033 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:51.190890    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:00:51.191033 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:51.190972    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:00:53.190010 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:53.189758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:53.190010 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:53.189780    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:53.190010 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:53.189884    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:53.190010 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:53.189896    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:00:53.190010 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:53.189978    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:00:53.190726 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:53.190047    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:00:55.190571 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:55.190527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:55.190571 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:55.190568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:55.191066 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:55.190567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:55.191066 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:55.190666    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:00:55.191066 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:55.190751    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:00:55.191066 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:55.190813    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:00:57.191097 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.190697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:57.191900 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.190716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:57.191900 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:57.191215    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:00:57.191900 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.190742    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:57.191900 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:57.191274    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:00:57.191900 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:57.191334    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:00:57.288664 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.288603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sr9hs" event={"ID":"678bccec-6cea-4a28-9f6f-390c8ffec2c1","Type":"ContainerStarted","Data":"1cda95205794c55e2735e99a3e76c54f8a5d0c4c05f8e89f076befce61dbcdce"}
Jul 29 19:00:57.291508 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.291466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal" event={"ID":"c90e4ebfb2b0c26370b1fc329ec87357","Type":"ContainerStarted","Data":"bb95c3ea385464d67e3fd7e50f29dfe121041f371d4bcaec9c7c899c29d4fe67"}
Jul 29 19:00:57.293935 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.293800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qpbzz" event={"ID":"7e80e4a8-0ff2-4412-a9d7-826cfce55e8c","Type":"ContainerStarted","Data":"6c9d2c3cb7afba418bebbd64f9f5abc94b11321f71c692f0faf2d4b3d8599134"}
Jul 29 19:00:57.296057 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.296022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7tb67" event={"ID":"841e2b86-5a6c-43d7-9919-da697a9ce802","Type":"ContainerStarted","Data":"798d1fe55ade775c2e70d41bb51a117dac61696e4385764223379e8f36081d07"}
Jul 29 19:00:57.299941 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.299905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz" event={"ID":"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f","Type":"ContainerStarted","Data":"69157bb5f6d53de7d16791f8f86bb879a35ba56ef6ed1f26598b1177eae931b6"}
Jul 29 19:00:57.300099 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.299985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz" event={"ID":"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f","Type":"ContainerStarted","Data":"8debf12a648ed0f032bc37f2663b57c707720e43bdf561051f3d3a41b741e4e8"}
Jul 29 19:00:57.300099 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.300021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz" event={"ID":"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f","Type":"ContainerStarted","Data":"09ccc60c6074b2c11b1eddc3af9c48fe9a1ba6ff50d3fddc2f78438cf9e90f09"}
Jul 29 19:00:57.300099 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.300068    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz" event={"ID":"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f","Type":"ContainerStarted","Data":"8019d3bde6e75308a82ef0f0d056fa95fb764b9616e935b37f6e4d046546bf32"}
Jul 29 19:00:57.300099 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.300096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz" event={"ID":"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f","Type":"ContainerStarted","Data":"e64339339c75d3507805579bce31331c18cb67f28298132b2b31920848d939ff"}
Jul 29 19:00:57.300336 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.300131    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz" event={"ID":"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f","Type":"ContainerStarted","Data":"ab500c4ba7e50f05c62e9654566f605e66bf6e2b5e7f2463adbb0d659c03bdff"}
Jul 29 19:00:57.303628 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.303598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l9btm" event={"ID":"fab1c53d-2ed6-48b8-86b6-6244542b8750","Type":"ContainerStarted","Data":"7ae3a372a63a1d9e46b684b55b1e10748eee3950d1582d05b33bcf725a333593"}
Jul 29 19:00:57.305261 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.305212    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sr9hs" podStartSLOduration=3.24542069 podStartE2EDuration="23.305174203s" totalImagesPullingTime="20.059753513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:57.304708107 +0000 UTC m=+23.604380609" watchObservedRunningTime="2026-07-29 19:00:57.305174203 +0000 UTC m=+23.604846705"
Jul 29 19:00:57.307375 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.307348    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" event={"ID":"8ad2b80255b384c7f34ba4b0599ef380","Type":"ContainerStarted","Data":"2f35f72e6278bb893155ac759a9b0b57711127907765af8663dd90406c33b8d2"}
Jul 29 19:00:57.309392 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.309366    2579 generic.go:356] "Generic (PLEG): container finished" podID="1acea82a-ad7e-4dfa-8737-c397979480fb" containerID="75d86a3b1ba910169e0b5b2af31173abefc59f5ec1f76929422943b1d26bee0b" exitCode=0
Jul 29 19:00:57.309504 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.309455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zm58d" event={"ID":"1acea82a-ad7e-4dfa-8737-c397979480fb","Type":"ContainerDied","Data":"75d86a3b1ba910169e0b5b2af31173abefc59f5ec1f76929422943b1d26bee0b"}
Jul 29 19:00:57.312733 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.312707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jmvb4" event={"ID":"6b657d2b-5b8b-401b-ba94-b2c15ca3720f","Type":"ContainerStarted","Data":"9f3db5e941503302e2ddb1f46587c6cafcd703d00a599b8cf80836719dcc1c94"}
Jul 29 19:00:57.315076 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.315045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb" event={"ID":"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09","Type":"ContainerStarted","Data":"fd8373dc5aceb2c464c0b9261afeca229c85675174254624ae01f05d20b4b084"}
Jul 29 19:00:57.318017 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.317958    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal" podStartSLOduration=22.317937277 podStartE2EDuration="22.317937277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:57.317136422 +0000 UTC m=+23.616808924" watchObservedRunningTime="2026-07-29 19:00:57.317937277 +0000 UTC m=+23.617609778"
Jul 29 19:00:57.347836 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.347768    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l9btm" podStartSLOduration=7.809210183 podStartE2EDuration="23.347744528s" totalImagesPullingTime="15.538534345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:57.331413038 +0000 UTC m=+23.631085540" watchObservedRunningTime="2026-07-29 19:00:57.347744528 +0000 UTC m=+23.647417026"
Jul 29 19:00:57.348100 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.348074    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7tb67" podStartSLOduration=7.872258185 podStartE2EDuration="23.348067214s" totalImagesPullingTime="15.475809029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:57.347715938 +0000 UTC m=+23.647388439" watchObservedRunningTime="2026-07-29 19:00:57.348067214 +0000 UTC m=+23.647739715"
Jul 29 19:00:57.364056 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.363987    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qpbzz" podStartSLOduration=2.9110491080000003 podStartE2EDuration="23.363964459s" totalImagesPullingTime="20.452915351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:57.363372784 +0000 UTC m=+23.663045286" watchObservedRunningTime="2026-07-29 19:00:57.363964459 +0000 UTC m=+23.663636957"
Jul 29 19:00:57.380602 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.380534    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jmvb4" podStartSLOduration=7.930138727 podStartE2EDuration="23.38051028s" totalImagesPullingTime="15.450371553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:57.379923787 +0000 UTC m=+23.679596287" watchObservedRunningTime="2026-07-29 19:00:57.38051028 +0000 UTC m=+23.680182785"
Jul 29 19:00:57.399574 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:57.398235    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" podStartSLOduration=22.398212973 podStartE2EDuration="22.398212973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:57.397921334 +0000 UTC m=+23.697593834" watchObservedRunningTime="2026-07-29 19:00:57.398212973 +0000 UTC m=+23.697885467"
Jul 29 19:00:58.075723 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:58.075686    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 19:00:58.107987 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:58.107861    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T19:00:58.075714327Z","UUID":"cfd1408c-e04d-4c0d-abc0-3e7ddd474c87","Handler":null,"Name":"","Endpoint":""}
Jul 29 19:00:58.109928 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:58.109904    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 19:00:58.110038 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:58.109961    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 19:00:58.320082 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:58.319982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mf8x6" event={"ID":"309ebf49-c66a-478e-af58-316c3d195af4","Type":"ContainerStarted","Data":"14ab2efa28a63b75afca49c12e039e3fa44b6ece85cf45c883e9f0d8c3f53763"}
Jul 29 19:00:58.322322 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:58.322285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb" event={"ID":"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09","Type":"ContainerStarted","Data":"552da2694d6f1102ad166886de85590e257d73998e00c567205d702270796d8c"}
Jul 29 19:00:58.334739 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:58.334690    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mf8x6" podStartSLOduration=3.551914429 podStartE2EDuration="24.334669417s" totalImagesPullingTime="20.782754988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:58.334268521 +0000 UTC m=+24.633941022" watchObservedRunningTime="2026-07-29 19:00:58.334669417 +0000 UTC m=+24.634341921"
Jul 29 19:00:59.190232 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:59.189903    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:00:59.190232 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:59.190052    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:00:59.190232 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:59.190133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:00:59.190232 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:59.190210    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:00:59.190720 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:00:59.190300    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:00:59.190720 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:00:59.190409    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:01:00.083494 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:00.083453    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jmvb4"
Jul 29 19:01:00.084144 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:00.084126    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jmvb4"
Jul 29 19:01:00.333689 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:00.333576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb" event={"ID":"e6d8a6bf-13b0-4efb-ab11-1472ef36dd09","Type":"ContainerStarted","Data":"1b7b6f72a4b63cc18a370736c14f34af2e8a5fb8b6701bd7064e05021e37d3eb"}
Jul 29 19:01:00.337803 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:00.337757    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz" event={"ID":"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f","Type":"ContainerStarted","Data":"421a8012084475cbc2341dfbdd2cefd55a3da57563dbb61c905fe784e9de42f2"}
Jul 29 19:01:00.337990 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:00.337854    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jmvb4"
Jul 29 19:01:00.338538 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:00.338517    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jmvb4"
Jul 29 19:01:00.351158 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:00.351073    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-94zmb" podStartSLOduration=3.551339578 podStartE2EDuration="26.351054701s" totalImagesPullingTime="22.799715123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:01:00.350677617 +0000 UTC m=+26.650350213" watchObservedRunningTime="2026-07-29 19:01:00.351054701 +0000 UTC m=+26.650727201"
Jul 29 19:01:01.190109 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:01.190064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:01:01.190578 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:01.190121    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:01:01.190578 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:01.190083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:01:01.190578 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:01.190223    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:01:01.190578 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:01.190304    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:01:01.190578 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:01.190395    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:01:02.191026 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:02.190723    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:01:02.191937 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:02.191165    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:01:02.346818 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:02.346772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz" event={"ID":"d0cc9ff8-42b8-4517-904b-c6a12d9fb37f","Type":"ContainerStarted","Data":"5bdac0f670f2d82fbfcf39d88840ed4b459d67e82fab2ecfa2a52526045fc10a"}
Jul 29 19:01:02.347105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:02.347086    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:01:02.349103 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:02.349070    2579 generic.go:356] "Generic (PLEG): container finished" podID="1acea82a-ad7e-4dfa-8737-c397979480fb" containerID="cc5e47e5dda3092484c3f6bfe04477c64c6af4535adbed2980352d53417ee858" exitCode=0
Jul 29 19:01:02.349268 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:02.349158    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zm58d" event={"ID":"1acea82a-ad7e-4dfa-8737-c397979480fb","Type":"ContainerDied","Data":"cc5e47e5dda3092484c3f6bfe04477c64c6af4535adbed2980352d53417ee858"}
Jul 29 19:01:02.365225 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:02.365180    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:01:02.379614 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:02.379521    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz" podStartSLOduration=7.729920901 podStartE2EDuration="28.379501608s" totalImagesPullingTime="20.649580707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:01:02.379119655 +0000 UTC m=+28.678792166" watchObservedRunningTime="2026-07-29 19:01:02.379501608 +0000 UTC m=+28.679174109"
Jul 29 19:01:03.190261 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:03.190183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:01:03.190579 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:03.190233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:01:03.190579 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:03.190400    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:01:03.190579 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:03.190416    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:01:03.351787 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:03.351747    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:01:03.352408 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:03.351848    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:01:03.369029 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:03.368970    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:01:03.841506 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:03.841179    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gsbwv"]
Jul 29 19:01:03.841750 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:03.841633    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:01:03.841804 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:03.841772    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:01:03.844584 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:03.844551    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gv68b"]
Jul 29 19:01:03.844769 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:03.844703    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:01:03.844839 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:03.844819    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:01:03.845599 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:03.845570    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zm2md"]
Jul 29 19:01:03.845747 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:03.845697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:01:03.845826 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:03.845801    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:01:05.190117 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:05.190073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:01:05.190562 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:05.190073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:01:05.190562 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:05.190252    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:01:05.190562 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:05.190279    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:01:06.190801 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:06.190699    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:01:06.191275 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:06.190831    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:01:06.909136 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:06.909017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:01:06.909136 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:06.909087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:01:06.909398 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:06.909212    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:01:06.909398 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:06.909215    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:01:06.909398 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:06.909292    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs podName:1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:38.909268806 +0000 UTC m=+65.208941285 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs") pod "network-metrics-daemon-zm2md" (UID: "1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:01:06.909398 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:06.909309    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret podName:898bc211-2d04-4fe7-b981-4ce7f9c0a856 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:38.909301611 +0000 UTC m=+65.208974089 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret") pod "global-pull-secret-syncer-gv68b" (UID: "898bc211-2d04-4fe7-b981-4ce7f9c0a856") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:01:07.009968 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:07.009924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drdp\" (UniqueName: \"kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp\") pod \"network-check-target-gsbwv\" (UID: \"07f5037f-a52e-4722-898b-8e3c53085d4b\") " pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:01:07.010149 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:07.010067    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:01:07.010149 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:07.010086    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:01:07.010149 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:07.010096    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4drdp for pod openshift-network-diagnostics/network-check-target-gsbwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:01:07.010322 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:07.010152    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp podName:07f5037f-a52e-4722-898b-8e3c53085d4b nodeName:}" failed. No retries permitted until 2026-07-29 19:01:39.010136096 +0000 UTC m=+65.309808579 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4drdp" (UniqueName: "kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp") pod "network-check-target-gsbwv" (UID: "07f5037f-a52e-4722-898b-8e3c53085d4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:01:07.190717 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:07.190617    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:01:07.190877 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:07.190623    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:01:07.190877 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:07.190740    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:01:07.190877 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:07.190821    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:01:08.193674 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:08.190796    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:01:08.193674 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:08.190993    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gv68b" podUID="898bc211-2d04-4fe7-b981-4ce7f9c0a856"
Jul 29 19:01:08.365184 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:08.365062    2579 generic.go:356] "Generic (PLEG): container finished" podID="1acea82a-ad7e-4dfa-8737-c397979480fb" containerID="3fb8c7acfed61676a87bc125bdf738b46c461a4f72faa601abd752bf22821e7f" exitCode=0
Jul 29 19:01:08.365184 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:08.365125    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zm58d" event={"ID":"1acea82a-ad7e-4dfa-8737-c397979480fb","Type":"ContainerDied","Data":"3fb8c7acfed61676a87bc125bdf738b46c461a4f72faa601abd752bf22821e7f"}
Jul 29 19:01:09.190726 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.190519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:01:09.190947 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.190551    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:01:09.190947 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:09.190801    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsbwv" podUID="07f5037f-a52e-4722-898b-8e3c53085d4b"
Jul 29 19:01:09.190947 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:09.190917    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:01:09.370524 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.370415    2579 generic.go:356] "Generic (PLEG): container finished" podID="1acea82a-ad7e-4dfa-8737-c397979480fb" containerID="ae7fd874a4784a352170edb5b80af0acb520fc9ce056675350134969b18e3e69" exitCode=0
Jul 29 19:01:09.370524 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.370467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zm58d" event={"ID":"1acea82a-ad7e-4dfa-8737-c397979480fb","Type":"ContainerDied","Data":"ae7fd874a4784a352170edb5b80af0acb520fc9ce056675350134969b18e3e69"}
Jul 29 19:01:09.481923 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.481896    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeReady"
Jul 29 19:01:09.482121 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.482035    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 19:01:09.528512 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.528480    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-79x22"]
Jul 29 19:01:09.531465 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.531441    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:09.534481 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.534455    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 19:01:09.534481 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.534466    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7dnhc\""
Jul 29 19:01:09.534670 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.534485    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 19:01:09.534769 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.534754    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qkzwn"]
Jul 29 19:01:09.537452 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.537426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qkzwn"
Jul 29 19:01:09.540474 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.540183    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 19:01:09.540474 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.540224    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 19:01:09.540474 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.540278    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-79x22"]
Jul 29 19:01:09.540474 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.540204    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 19:01:09.540474 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.540337    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2wk54\""
Jul 29 19:01:09.546292 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.546269    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qkzwn"]
Jul 29 19:01:09.629813 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.629698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/614641fc-597a-4b94-ba7c-33f078d5ed1c-cert\") pod \"ingress-canary-qkzwn\" (UID: \"614641fc-597a-4b94-ba7c-33f078d5ed1c\") " pod="openshift-ingress-canary/ingress-canary-qkzwn"
Jul 29 19:01:09.629813 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.629751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z78dw\" (UniqueName: \"kubernetes.io/projected/15228df5-30d8-4cd8-8de4-a55d961a28d8-kube-api-access-z78dw\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:09.629813 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.629780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq8pt\" (UniqueName: \"kubernetes.io/projected/614641fc-597a-4b94-ba7c-33f078d5ed1c-kube-api-access-fq8pt\") pod \"ingress-canary-qkzwn\" (UID: \"614641fc-597a-4b94-ba7c-33f078d5ed1c\") " pod="openshift-ingress-canary/ingress-canary-qkzwn"
Jul 29 19:01:09.630086 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.629824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:09.630086 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.629871    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15228df5-30d8-4cd8-8de4-a55d961a28d8-tmp-dir\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:09.630086 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.629911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/15228df5-30d8-4cd8-8de4-a55d961a28d8-config-volume\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:09.731080 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.731033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/614641fc-597a-4b94-ba7c-33f078d5ed1c-cert\") pod \"ingress-canary-qkzwn\" (UID: \"614641fc-597a-4b94-ba7c-33f078d5ed1c\") " pod="openshift-ingress-canary/ingress-canary-qkzwn"
Jul 29 19:01:09.731330 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.731086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z78dw\" (UniqueName: \"kubernetes.io/projected/15228df5-30d8-4cd8-8de4-a55d961a28d8-kube-api-access-z78dw\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:09.731330 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.731113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq8pt\" (UniqueName: \"kubernetes.io/projected/614641fc-597a-4b94-ba7c-33f078d5ed1c-kube-api-access-fq8pt\") pod \"ingress-canary-qkzwn\" (UID: \"614641fc-597a-4b94-ba7c-33f078d5ed1c\") " pod="openshift-ingress-canary/ingress-canary-qkzwn"
Jul 29 19:01:09.731330 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.731152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:09.731330 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:09.731279    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:01:09.731330 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.731284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15228df5-30d8-4cd8-8de4-a55d961a28d8-tmp-dir\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:09.731582 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:09.731347    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls podName:15228df5-30d8-4cd8-8de4-a55d961a28d8 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:10.231326364 +0000 UTC m=+36.530998855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls") pod "dns-default-79x22" (UID: "15228df5-30d8-4cd8-8de4-a55d961a28d8") : secret "dns-default-metrics-tls" not found
Jul 29 19:01:09.731582 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.731393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/15228df5-30d8-4cd8-8de4-a55d961a28d8-config-volume\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:09.731689 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.731658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15228df5-30d8-4cd8-8de4-a55d961a28d8-tmp-dir\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:09.731953 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.731931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/15228df5-30d8-4cd8-8de4-a55d961a28d8-config-volume\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:09.735964 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.735941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/614641fc-597a-4b94-ba7c-33f078d5ed1c-cert\") pod \"ingress-canary-qkzwn\" (UID: \"614641fc-597a-4b94-ba7c-33f078d5ed1c\") " pod="openshift-ingress-canary/ingress-canary-qkzwn"
Jul 29 19:01:09.740475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.740443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq8pt\" (UniqueName: \"kubernetes.io/projected/614641fc-597a-4b94-ba7c-33f078d5ed1c-kube-api-access-fq8pt\") pod \"ingress-canary-qkzwn\" (UID: \"614641fc-597a-4b94-ba7c-33f078d5ed1c\") " pod="openshift-ingress-canary/ingress-canary-qkzwn"
Jul 29 19:01:09.740917 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.740898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z78dw\" (UniqueName: \"kubernetes.io/projected/15228df5-30d8-4cd8-8de4-a55d961a28d8-kube-api-access-z78dw\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:09.848313 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:09.848274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qkzwn"
Jul 29 19:01:10.000596 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:10.000555    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qkzwn"]
Jul 29 19:01:10.004564 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:01:10.004529    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614641fc_597a_4b94_ba7c_33f078d5ed1c.slice/crio-6ec3625bee4b1a27d31af1262e7e91129e5a20d52369cd7404242cd7b0d107cf WatchSource:0}: Error finding container 6ec3625bee4b1a27d31af1262e7e91129e5a20d52369cd7404242cd7b0d107cf: Status 404 returned error can't find the container with id 6ec3625bee4b1a27d31af1262e7e91129e5a20d52369cd7404242cd7b0d107cf
Jul 29 19:01:10.190263 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:10.190156    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:01:10.193447 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:10.193413    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 19:01:10.236213 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:10.236146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:10.236419 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:10.236346    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:01:10.236472 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:10.236438    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls podName:15228df5-30d8-4cd8-8de4-a55d961a28d8 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:11.236414525 +0000 UTC m=+37.536087229 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls") pod "dns-default-79x22" (UID: "15228df5-30d8-4cd8-8de4-a55d961a28d8") : secret "dns-default-metrics-tls" not found
Jul 29 19:01:10.374882 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:10.374839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qkzwn" event={"ID":"614641fc-597a-4b94-ba7c-33f078d5ed1c","Type":"ContainerStarted","Data":"6ec3625bee4b1a27d31af1262e7e91129e5a20d52369cd7404242cd7b0d107cf"}
Jul 29 19:01:11.190232 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:11.190177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:01:11.190475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:11.190247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:01:11.193096 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:11.193068    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 19:01:11.194465 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:11.194435    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 19:01:11.194465 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:11.194461    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qs6c5\""
Jul 29 19:01:11.194670 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:11.194517    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 19:01:11.194670 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:11.194435    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qh2tz\""
Jul 29 19:01:11.244486 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:11.244438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:11.244735 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:11.244625    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:01:11.244735 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:11.244727    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls podName:15228df5-30d8-4cd8-8de4-a55d961a28d8 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:13.244696266 +0000 UTC m=+39.544368749 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls") pod "dns-default-79x22" (UID: "15228df5-30d8-4cd8-8de4-a55d961a28d8") : secret "dns-default-metrics-tls" not found
Jul 29 19:01:13.261493 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:13.261173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:13.262079 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:13.261368    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:01:13.262079 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:13.261689    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls podName:15228df5-30d8-4cd8-8de4-a55d961a28d8 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:17.261657846 +0000 UTC m=+43.561330325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls") pod "dns-default-79x22" (UID: "15228df5-30d8-4cd8-8de4-a55d961a28d8") : secret "dns-default-metrics-tls" not found
Jul 29 19:01:15.394033 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:15.393974    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qkzwn" event={"ID":"614641fc-597a-4b94-ba7c-33f078d5ed1c","Type":"ContainerStarted","Data":"10bd8f42debdde2c5bb388e6ac41d70270d96cb598ac31e98c45aacf3d885c47"}
Jul 29 19:01:15.397047 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:15.397001    2579 generic.go:356] "Generic (PLEG): container finished" podID="1acea82a-ad7e-4dfa-8737-c397979480fb" containerID="fd9564b5a3e75399970d44bee24789f311e5a54d098477ed7105963d3f53b348" exitCode=0
Jul 29 19:01:15.397207 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:15.397057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zm58d" event={"ID":"1acea82a-ad7e-4dfa-8737-c397979480fb","Type":"ContainerDied","Data":"fd9564b5a3e75399970d44bee24789f311e5a54d098477ed7105963d3f53b348"}
Jul 29 19:01:15.410274 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:15.410219    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qkzwn" podStartSLOduration=1.766552745 podStartE2EDuration="6.410180822s" totalImagesPullingTime="4.643628077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:01:15.409624288 +0000 UTC m=+41.709296789" watchObservedRunningTime="2026-07-29 19:01:15.410180822 +0000 UTC m=+41.709853322"
Jul 29 19:01:16.403655 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:16.403613    2579 generic.go:356] "Generic (PLEG): container finished" podID="1acea82a-ad7e-4dfa-8737-c397979480fb" containerID="f71b8e81679711d9212631b83cd82e058e44bc8b84dd70fec24e02d8e519d27a" exitCode=0
Jul 29 19:01:16.404226 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:16.403695    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zm58d" event={"ID":"1acea82a-ad7e-4dfa-8737-c397979480fb","Type":"ContainerDied","Data":"f71b8e81679711d9212631b83cd82e058e44bc8b84dd70fec24e02d8e519d27a"}
Jul 29 19:01:17.295396 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:17.295342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:17.295641 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:17.295518    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:01:17.295641 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:17.295603    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls podName:15228df5-30d8-4cd8-8de4-a55d961a28d8 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:25.295581728 +0000 UTC m=+51.595254207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls") pod "dns-default-79x22" (UID: "15228df5-30d8-4cd8-8de4-a55d961a28d8") : secret "dns-default-metrics-tls" not found
Jul 29 19:01:17.410057 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:17.410012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zm58d" event={"ID":"1acea82a-ad7e-4dfa-8737-c397979480fb","Type":"ContainerStarted","Data":"fc3e952f9cb88a67ad5afc02ae905960a48159d4bbc6abbf1256134e991d3616"}
Jul 29 19:01:17.434475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:17.434416    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zm58d" podStartSLOduration=6.731279883 podStartE2EDuration="43.434398858s" totalImagesPullingTime="36.703118975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:01:17.432432282 +0000 UTC m=+43.732104783" watchObservedRunningTime="2026-07-29 19:01:17.434398858 +0000 UTC m=+43.734071359"
Jul 29 19:01:25.357889 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:25.357837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:25.358482 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:25.358022    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:01:25.358482 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:25.358113    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls podName:15228df5-30d8-4cd8-8de4-a55d961a28d8 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:41.358090772 +0000 UTC m=+67.657763267 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls") pod "dns-default-79x22" (UID: "15228df5-30d8-4cd8-8de4-a55d961a28d8") : secret "dns-default-metrics-tls" not found
Jul 29 19:01:27.092980 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.092939    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b46dcb55c-q4dhq"]
Jul 29 19:01:27.108728 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.108686    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b46dcb55c-q4dhq"]
Jul 29 19:01:27.108970 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.108839    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.111971 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.111945    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 19:01:27.112148 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.111948    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 19:01:27.112148 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.112095    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xdl87\""
Jul 29 19:01:27.112298 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.112161    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 19:01:27.120773 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.120737    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 19:01:27.273581 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.273524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6fcf51f-c473-441e-8136-e5c300f7dc65-image-registry-private-configuration\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.273810 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.273610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf51f-c473-441e-8136-e5c300f7dc65-ca-trust-extracted\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.273810 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.273651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.273810 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.273677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-certificates\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.273810 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.273700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6fcf51f-c473-441e-8136-e5c300f7dc65-trusted-ca\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.273810 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.273736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6fcf51f-c473-441e-8136-e5c300f7dc65-installation-pull-secrets\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.273968 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.273830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-697hr\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-kube-api-access-697hr\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.273968 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.273876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-bound-sa-token\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.374819 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.374717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6fcf51f-c473-441e-8136-e5c300f7dc65-installation-pull-secrets\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.374819 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.374772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-697hr\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-kube-api-access-697hr\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.374819 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.374808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-bound-sa-token\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.375088 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.374831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6fcf51f-c473-441e-8136-e5c300f7dc65-image-registry-private-configuration\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.375088 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.374888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf51f-c473-441e-8136-e5c300f7dc65-ca-trust-extracted\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.375088 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.375045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.375269 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.375102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-certificates\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.375269 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.375131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6fcf51f-c473-441e-8136-e5c300f7dc65-trusted-ca\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.375269 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:27.375251    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 19:01:27.375403 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:27.375277    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46dcb55c-q4dhq: secret "image-registry-tls" not found
Jul 29 19:01:27.375403 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:27.375375    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls podName:d6fcf51f-c473-441e-8136-e5c300f7dc65 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:27.875345988 +0000 UTC m=+54.175018487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls") pod "image-registry-5b46dcb55c-q4dhq" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65") : secret "image-registry-tls" not found
Jul 29 19:01:27.375485 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.375462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf51f-c473-441e-8136-e5c300f7dc65-ca-trust-extracted\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.376204 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.376164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-certificates\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.376353 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.376215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6fcf51f-c473-441e-8136-e5c300f7dc65-trusted-ca\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.378766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.378736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6fcf51f-c473-441e-8136-e5c300f7dc65-image-registry-private-configuration\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.378891 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.378737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6fcf51f-c473-441e-8136-e5c300f7dc65-installation-pull-secrets\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.384941 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.384873    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-bound-sa-token\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.385073 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.384920    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-697hr\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-kube-api-access-697hr\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.878419 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:27.878375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:27.878644 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:27.878547    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 19:01:27.878644 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:27.878571    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46dcb55c-q4dhq: secret "image-registry-tls" not found
Jul 29 19:01:27.878714 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:27.878648    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls podName:d6fcf51f-c473-441e-8136-e5c300f7dc65 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:28.878627824 +0000 UTC m=+55.178300303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls") pod "image-registry-5b46dcb55c-q4dhq" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65") : secret "image-registry-tls" not found
Jul 29 19:01:28.886420 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:28.886373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:28.886897 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:28.886538    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 19:01:28.886897 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:28.886559    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46dcb55c-q4dhq: secret "image-registry-tls" not found
Jul 29 19:01:28.886897 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:28.886630    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls podName:d6fcf51f-c473-441e-8136-e5c300f7dc65 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:30.886610154 +0000 UTC m=+57.186282636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls") pod "image-registry-5b46dcb55c-q4dhq" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65") : secret "image-registry-tls" not found
Jul 29 19:01:30.903088 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:30.903009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:30.903608 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:30.903172    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 19:01:30.903608 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:30.903214    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46dcb55c-q4dhq: secret "image-registry-tls" not found
Jul 29 19:01:30.903608 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:30.903278    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls podName:d6fcf51f-c473-441e-8136-e5c300f7dc65 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:34.903260731 +0000 UTC m=+61.202933212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls") pod "image-registry-5b46dcb55c-q4dhq" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65") : secret "image-registry-tls" not found
Jul 29 19:01:34.935388 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:34.935334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:34.935879 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:34.935554    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 19:01:34.935879 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:34.935578    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46dcb55c-q4dhq: secret "image-registry-tls" not found
Jul 29 19:01:34.935879 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:34.935657    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls podName:d6fcf51f-c473-441e-8136-e5c300f7dc65 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:42.935636679 +0000 UTC m=+69.235309162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls") pod "image-registry-5b46dcb55c-q4dhq" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65") : secret "image-registry-tls" not found
Jul 29 19:01:35.372438 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:35.372329    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pglrz"
Jul 29 19:01:38.967613 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:38.967560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:01:38.968067 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:38.967653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:01:38.970735 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:38.970693    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 19:01:38.970735 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:38.970700    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 19:01:38.978394 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:38.978345    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 19:01:38.978511 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:38.978475    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs podName:1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42 nodeName:}" failed. No retries permitted until 2026-07-29 19:02:42.978446274 +0000 UTC m=+129.278118754 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs") pod "network-metrics-daemon-zm2md" (UID: "1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42") : secret "metrics-daemon-secret" not found
Jul 29 19:01:38.981035 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:38.981005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/898bc211-2d04-4fe7-b981-4ce7f9c0a856-original-pull-secret\") pod \"global-pull-secret-syncer-gv68b\" (UID: \"898bc211-2d04-4fe7-b981-4ce7f9c0a856\") " pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:01:39.003138 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:39.003089    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gv68b"
Jul 29 19:01:39.069521 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:39.068649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drdp\" (UniqueName: \"kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp\") pod \"network-check-target-gsbwv\" (UID: \"07f5037f-a52e-4722-898b-8e3c53085d4b\") " pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:01:39.081971 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:39.081661    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 19:01:39.085350 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:39.085025    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 19:01:39.098173 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:39.098077    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4drdp\" (UniqueName: \"kubernetes.io/projected/07f5037f-a52e-4722-898b-8e3c53085d4b-kube-api-access-4drdp\") pod \"network-check-target-gsbwv\" (UID: \"07f5037f-a52e-4722-898b-8e3c53085d4b\") " pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:01:39.107438 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:39.107405    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qs6c5\""
Jul 29 19:01:39.115556 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:39.115471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:01:39.167025 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:39.166973    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gv68b"]
Jul 29 19:01:39.176529 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:01:39.176456    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898bc211_2d04_4fe7_b981_4ce7f9c0a856.slice/crio-ffaa7ade4056ac2dd04ae6bcd16e7742bf9a0acdda585eb5072140e4db28378c WatchSource:0}: Error finding container ffaa7ade4056ac2dd04ae6bcd16e7742bf9a0acdda585eb5072140e4db28378c: Status 404 returned error can't find the container with id ffaa7ade4056ac2dd04ae6bcd16e7742bf9a0acdda585eb5072140e4db28378c
Jul 29 19:01:39.258999 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:39.258902    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gsbwv"]
Jul 29 19:01:39.262505 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:01:39.262476    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f5037f_a52e_4722_898b_8e3c53085d4b.slice/crio-a8719650bc14a7a465e3de312cb5bf7e6bbbf2a9921dfaeae9e728bfdbcb75d4 WatchSource:0}: Error finding container a8719650bc14a7a465e3de312cb5bf7e6bbbf2a9921dfaeae9e728bfdbcb75d4: Status 404 returned error can't find the container with id a8719650bc14a7a465e3de312cb5bf7e6bbbf2a9921dfaeae9e728bfdbcb75d4
Jul 29 19:01:39.470496 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:39.470449    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gsbwv" event={"ID":"07f5037f-a52e-4722-898b-8e3c53085d4b","Type":"ContainerStarted","Data":"a8719650bc14a7a465e3de312cb5bf7e6bbbf2a9921dfaeae9e728bfdbcb75d4"}
Jul 29 19:01:39.471699 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:39.471664    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gv68b" event={"ID":"898bc211-2d04-4fe7-b981-4ce7f9c0a856","Type":"ContainerStarted","Data":"ffaa7ade4056ac2dd04ae6bcd16e7742bf9a0acdda585eb5072140e4db28378c"}
Jul 29 19:01:41.389629 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:41.389579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:01:41.390251 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:41.389821    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:01:41.390251 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:41.389893    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls podName:15228df5-30d8-4cd8-8de4-a55d961a28d8 nodeName:}" failed. No retries permitted until 2026-07-29 19:02:13.389871436 +0000 UTC m=+99.689543936 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls") pod "dns-default-79x22" (UID: "15228df5-30d8-4cd8-8de4-a55d961a28d8") : secret "dns-default-metrics-tls" not found
Jul 29 19:01:43.004024 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:43.003967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:43.004533 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:43.004202    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 19:01:43.004533 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:43.004231    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46dcb55c-q4dhq: secret "image-registry-tls" not found
Jul 29 19:01:43.004533 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:43.004317    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls podName:d6fcf51f-c473-441e-8136-e5c300f7dc65 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:59.004291373 +0000 UTC m=+85.303963857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls") pod "image-registry-5b46dcb55c-q4dhq" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65") : secret "image-registry-tls" not found
Jul 29 19:01:44.487369 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:44.487325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gsbwv" event={"ID":"07f5037f-a52e-4722-898b-8e3c53085d4b","Type":"ContainerStarted","Data":"38598d9adaa0cb3f417d0a0fdd03f1734f4d007a673348dc9d58686382b4cb73"}
Jul 29 19:01:44.487845 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:44.487447    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:01:44.488809 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:44.488786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gv68b" event={"ID":"898bc211-2d04-4fe7-b981-4ce7f9c0a856","Type":"ContainerStarted","Data":"f986050289d4af4977ad0780837a71bc35772a4dc1fb04087f2de9fc19874183"}
Jul 29 19:01:44.504320 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:44.504261    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gsbwv" podStartSLOduration=66.085248726 podStartE2EDuration="1m10.504244337s" totalImagesPullingTime="4.418995611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:01:44.503299864 +0000 UTC m=+70.802972368" watchObservedRunningTime="2026-07-29 19:01:44.504244337 +0000 UTC m=+70.803916927"
Jul 29 19:01:44.518791 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:44.518732    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gv68b" podStartSLOduration=66.011077535 podStartE2EDuration="1m10.518713998s" totalImagesPullingTime="4.507636463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:01:44.517733289 +0000 UTC m=+70.817405819" watchObservedRunningTime="2026-07-29 19:01:44.518713998 +0000 UTC m=+70.818386533"
Jul 29 19:01:59.021929 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:01:59.021780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:01:59.021929 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:59.021923    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 19:01:59.022472 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:59.021943    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46dcb55c-q4dhq: secret "image-registry-tls" not found
Jul 29 19:01:59.022472 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:01:59.022004    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls podName:d6fcf51f-c473-441e-8136-e5c300f7dc65 nodeName:}" failed. No retries permitted until 2026-07-29 19:02:31.021989186 +0000 UTC m=+117.321661665 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls") pod "image-registry-5b46dcb55c-q4dhq" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65") : secret "image-registry-tls" not found
Jul 29 19:02:13.427946 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:02:13.427909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:02:13.428287 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:02:13.428073    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:02:13.428287 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:02:13.428142    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls podName:15228df5-30d8-4cd8-8de4-a55d961a28d8 nodeName:}" failed. No retries permitted until 2026-07-29 19:03:17.428125754 +0000 UTC m=+163.727798237 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls") pod "dns-default-79x22" (UID: "15228df5-30d8-4cd8-8de4-a55d961a28d8") : secret "dns-default-metrics-tls" not found
Jul 29 19:02:15.493639 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:02:15.493609    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gsbwv"
Jul 29 19:02:31.053285 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:02:31.053239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:02:31.053766 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:02:31.053376    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 19:02:31.053766 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:02:31.053390    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46dcb55c-q4dhq: secret "image-registry-tls" not found
Jul 29 19:02:31.053766 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:02:31.053453    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls podName:d6fcf51f-c473-441e-8136-e5c300f7dc65 nodeName:}" failed. No retries permitted until 2026-07-29 19:03:35.053438168 +0000 UTC m=+181.353110646 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls") pod "image-registry-5b46dcb55c-q4dhq" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65") : secret "image-registry-tls" not found
Jul 29 19:02:43.036728 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:02:43.036668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:02:43.037117 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:02:43.036822    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 19:02:43.037117 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:02:43.036891    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs podName:1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:45.036874384 +0000 UTC m=+251.336546862 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs") pod "network-metrics-daemon-zm2md" (UID: "1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42") : secret "metrics-daemon-secret" not found
Jul 29 19:03:12.541528 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:03:12.541470    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-79x22" podUID="15228df5-30d8-4cd8-8de4-a55d961a28d8"
Jul 29 19:03:12.695869 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:03:12.695839    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-79x22"
Jul 29 19:03:14.212205 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:03:14.212155    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:03:17.484870 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:03:17.484817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:03:17.485263 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:03:17.484969    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:03:17.485263 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:03:17.485037    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls podName:15228df5-30d8-4cd8-8de4-a55d961a28d8 nodeName:}" failed. No retries permitted until 2026-07-29 19:05:19.48502267 +0000 UTC m=+285.784695152 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls") pod "dns-default-79x22" (UID: "15228df5-30d8-4cd8-8de4-a55d961a28d8") : secret "dns-default-metrics-tls" not found
Jul 29 19:03:25.190600 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:03:25.190515    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:03:30.122314 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:03:30.122256    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq" podUID="d6fcf51f-c473-441e-8136-e5c300f7dc65"
Jul 29 19:03:30.739390 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:03:30.739344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:03:35.112046 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:03:35.111997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls\") pod \"image-registry-5b46dcb55c-q4dhq\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") " pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:03:35.112464 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:03:35.112156    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 19:03:35.112464 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:03:35.112175    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b46dcb55c-q4dhq: secret "image-registry-tls" not found
Jul 29 19:03:35.112464 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:03:35.112262    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls podName:d6fcf51f-c473-441e-8136-e5c300f7dc65 nodeName:}" failed. No retries permitted until 2026-07-29 19:05:37.112244561 +0000 UTC m=+303.411917039 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls") pod "image-registry-5b46dcb55c-q4dhq" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65") : secret "image-registry-tls" not found
Jul 29 19:04:39.461592 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.461551    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p2dkv"]
Jul 29 19:04:39.464489 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.464468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p2dkv"
Jul 29 19:04:39.466628 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.466599    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld"]
Jul 29 19:04:39.467234 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.467218    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 29 19:04:39.467320 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.467221    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lqbw9\""
Jul 29 19:04:39.467599 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.467580    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:04:39.469295 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.469279    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"]
Jul 29 19:04:39.469410 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.469392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld"
Jul 29 19:04:39.472094 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.472073    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569"]
Jul 29 19:04:39.472183 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.472168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"
Jul 29 19:04:39.472276 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.472220    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 29 19:04:39.472397 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.472381    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xsg2b\""
Jul 29 19:04:39.472483 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.472397    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 29 19:04:39.473320 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.473301    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 29 19:04:39.473422 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.473326    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:04:39.474727 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.474711    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:04:39.474806 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.474777    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p2dkv"]
Jul 29 19:04:39.474870 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.474857    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569"
Jul 29 19:04:39.475431 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.475416    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2lql5\""
Jul 29 19:04:39.475525 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.475451    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 29 19:04:39.475756 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.475737    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 29 19:04:39.477319 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.477300    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 29 19:04:39.477414 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.477377    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 29 19:04:39.478000 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.477615    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 29 19:04:39.478000 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.477625    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-h6jhm\""
Jul 29 19:04:39.478000 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.477670    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:04:39.484073 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.484057    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld"]
Jul 29 19:04:39.484918 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.484900    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569"]
Jul 29 19:04:39.488326 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.488294    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"]
Jul 29 19:04:39.557991 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.557965    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-45jh7"]
Jul 29 19:04:39.560882 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.560858    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.561855 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.561835    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"]
Jul 29 19:04:39.564360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.564342    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-d52sf"]
Jul 29 19:04:39.564474 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.564457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:04:39.564962 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.564942    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 19:04:39.565107 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.565019    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fw26q\""
Jul 29 19:04:39.565181 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.565138    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 29 19:04:39.565276 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.565257    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 29 19:04:39.565659 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.565642    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 19:04:39.566374 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.566356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh7qv\" (UniqueName: \"kubernetes.io/projected/185b4d06-9bb4-4f51-a4e3-6a156fc3fef2-kube-api-access-jh7qv\") pod \"volume-data-source-validator-549c6d54cf-p2dkv\" (UID: \"185b4d06-9bb4-4f51-a4e3-6a156fc3fef2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p2dkv"
Jul 29 19:04:39.566461 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.566383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxc92\" (UniqueName: \"kubernetes.io/projected/301cadcb-6dd5-4f64-b02c-bfaf77576609-kube-api-access-kxc92\") pod \"cluster-samples-operator-6748489656-gfsnv\" (UID: \"301cadcb-6dd5-4f64-b02c-bfaf77576609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"
Jul 29 19:04:39.566461 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.566404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfjr9\" (UniqueName: \"kubernetes.io/projected/30cfb52e-9435-492b-b39d-f6ab737038bd-kube-api-access-jfjr9\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld\" (UID: \"30cfb52e-9435-492b-b39d-f6ab737038bd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld"
Jul 29 19:04:39.566461 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.566421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30cfb52e-9435-492b-b39d-f6ab737038bd-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld\" (UID: \"30cfb52e-9435-492b-b39d-f6ab737038bd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld"
Jul 29 19:04:39.566461 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.566436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30cfb52e-9435-492b-b39d-f6ab737038bd-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld\" (UID: \"30cfb52e-9435-492b-b39d-f6ab737038bd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld"
Jul 29 19:04:39.566647 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.566533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqf7w\" (UniqueName: \"kubernetes.io/projected/1a486fe1-02df-4c80-8b23-83cb3e070a5a-kube-api-access-qqf7w\") pod \"service-ca-operator-5d6c75f84f-ds569\" (UID: \"1a486fe1-02df-4c80-8b23-83cb3e070a5a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569"
Jul 29 19:04:39.566647 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.566564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-gfsnv\" (UID: \"301cadcb-6dd5-4f64-b02c-bfaf77576609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"
Jul 29 19:04:39.566741 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.566639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a486fe1-02df-4c80-8b23-83cb3e070a5a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ds569\" (UID: \"1a486fe1-02df-4c80-8b23-83cb3e070a5a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569"
Jul 29 19:04:39.566741 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.566672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a486fe1-02df-4c80-8b23-83cb3e070a5a-config\") pod \"service-ca-operator-5d6c75f84f-ds569\" (UID: \"1a486fe1-02df-4c80-8b23-83cb3e070a5a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569"
Jul 29 19:04:39.567030 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.567011    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 29 19:04:39.567257 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.567171    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:39.567702 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.567681    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 19:04:39.569446 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.569425    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vg5t9\""
Jul 29 19:04:39.569537 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.569489    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 29 19:04:39.569590 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.569571    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 19:04:39.572505 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.572488    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:04:39.572629 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.572607    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 29 19:04:39.572739 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.572634    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 29 19:04:39.572739 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.572721    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 29 19:04:39.573050 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.573035    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 29 19:04:39.573166 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.573072    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-h9249\""
Jul 29 19:04:39.577721 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.577693    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-45jh7"]
Jul 29 19:04:39.580904 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.580884    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"]
Jul 29 19:04:39.582895 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.582875    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 29 19:04:39.583164 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.583147    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-d52sf"]
Jul 29 19:04:39.667598 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.667572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjr9\" (UniqueName: \"kubernetes.io/projected/30cfb52e-9435-492b-b39d-f6ab737038bd-kube-api-access-jfjr9\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld\" (UID: \"30cfb52e-9435-492b-b39d-f6ab737038bd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld"
Jul 29 19:04:39.667737 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.667605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68123628-c31f-4b00-b091-0c1e8802972a-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.667737 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.667628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30cfb52e-9435-492b-b39d-f6ab737038bd-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld\" (UID: \"30cfb52e-9435-492b-b39d-f6ab737038bd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld"
Jul 29 19:04:39.667737 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.667680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30cfb52e-9435-492b-b39d-f6ab737038bd-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld\" (UID: \"30cfb52e-9435-492b-b39d-f6ab737038bd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld"
Jul 29 19:04:39.667737 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.667710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a622c80f-34c3-4f10-97ca-e056df858797-config\") pod \"console-operator-584dcb6b54-d52sf\" (UID: \"a622c80f-34c3-4f10-97ca-e056df858797\") " pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:39.667944 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.667741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/68123628-c31f-4b00-b091-0c1e8802972a-snapshots\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.667944 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.667806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqf7w\" (UniqueName: \"kubernetes.io/projected/1a486fe1-02df-4c80-8b23-83cb3e070a5a-kube-api-access-qqf7w\") pod \"service-ca-operator-5d6c75f84f-ds569\" (UID: \"1a486fe1-02df-4c80-8b23-83cb3e070a5a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569"
Jul 29 19:04:39.667944 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.667859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a486fe1-02df-4c80-8b23-83cb3e070a5a-config\") pod \"service-ca-operator-5d6c75f84f-ds569\" (UID: \"1a486fe1-02df-4c80-8b23-83cb3e070a5a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569"
Jul 29 19:04:39.667944 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.667888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68123628-c31f-4b00-b091-0c1e8802972a-tmp\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.667944 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.667920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bv5r\" (UniqueName: \"kubernetes.io/projected/a622c80f-34c3-4f10-97ca-e056df858797-kube-api-access-9bv5r\") pod \"console-operator-584dcb6b54-d52sf\" (UID: \"a622c80f-34c3-4f10-97ca-e056df858797\") " pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:39.668181 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.667995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxc92\" (UniqueName: \"kubernetes.io/projected/301cadcb-6dd5-4f64-b02c-bfaf77576609-kube-api-access-kxc92\") pod \"cluster-samples-operator-6748489656-gfsnv\" (UID: \"301cadcb-6dd5-4f64-b02c-bfaf77576609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"
Jul 29 19:04:39.668181 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.668039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:04:39.668181 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.668080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a622c80f-34c3-4f10-97ca-e056df858797-trusted-ca\") pod \"console-operator-584dcb6b54-d52sf\" (UID: \"a622c80f-34c3-4f10-97ca-e056df858797\") " pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:39.668181 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.668107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68123628-c31f-4b00-b091-0c1e8802972a-service-ca-bundle\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.668181 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.668130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a622c80f-34c3-4f10-97ca-e056df858797-serving-cert\") pod \"console-operator-584dcb6b54-d52sf\" (UID: \"a622c80f-34c3-4f10-97ca-e056df858797\") " pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:39.668181 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.668156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-gfsnv\" (UID: \"301cadcb-6dd5-4f64-b02c-bfaf77576609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"
Jul 29 19:04:39.668482 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.668179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvv7j\" (UniqueName: \"kubernetes.io/projected/68123628-c31f-4b00-b091-0c1e8802972a-kube-api-access-rvv7j\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.668482 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.668228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:04:39.668482 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:39.668245    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 19:04:39.668482 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.668257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ffvj\" (UniqueName: \"kubernetes.io/projected/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-kube-api-access-2ffvj\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:04:39.668482 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.668283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a486fe1-02df-4c80-8b23-83cb3e070a5a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ds569\" (UID: \"1a486fe1-02df-4c80-8b23-83cb3e070a5a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569"
Jul 29 19:04:39.668482 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.668321    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30cfb52e-9435-492b-b39d-f6ab737038bd-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld\" (UID: \"30cfb52e-9435-492b-b39d-f6ab737038bd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld"
Jul 29 19:04:39.668482 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.668339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh7qv\" (UniqueName: \"kubernetes.io/projected/185b4d06-9bb4-4f51-a4e3-6a156fc3fef2-kube-api-access-jh7qv\") pod \"volume-data-source-validator-549c6d54cf-p2dkv\" (UID: \"185b4d06-9bb4-4f51-a4e3-6a156fc3fef2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p2dkv"
Jul 29 19:04:39.668482 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.668363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68123628-c31f-4b00-b091-0c1e8802972a-serving-cert\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.668482 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:39.668419    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls podName:301cadcb-6dd5-4f64-b02c-bfaf77576609 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:40.168405735 +0000 UTC m=+246.468078213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls") pod "cluster-samples-operator-6748489656-gfsnv" (UID: "301cadcb-6dd5-4f64-b02c-bfaf77576609") : secret "samples-operator-tls" not found
Jul 29 19:04:39.668934 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.668506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a486fe1-02df-4c80-8b23-83cb3e070a5a-config\") pod \"service-ca-operator-5d6c75f84f-ds569\" (UID: \"1a486fe1-02df-4c80-8b23-83cb3e070a5a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569"
Jul 29 19:04:39.670003 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.669983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30cfb52e-9435-492b-b39d-f6ab737038bd-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld\" (UID: \"30cfb52e-9435-492b-b39d-f6ab737038bd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld"
Jul 29 19:04:39.670410 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.670389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a486fe1-02df-4c80-8b23-83cb3e070a5a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ds569\" (UID: \"1a486fe1-02df-4c80-8b23-83cb3e070a5a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569"
Jul 29 19:04:39.677334 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.677310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqf7w\" (UniqueName: \"kubernetes.io/projected/1a486fe1-02df-4c80-8b23-83cb3e070a5a-kube-api-access-qqf7w\") pod \"service-ca-operator-5d6c75f84f-ds569\" (UID: \"1a486fe1-02df-4c80-8b23-83cb3e070a5a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569"
Jul 29 19:04:39.677435 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.677403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfjr9\" (UniqueName: \"kubernetes.io/projected/30cfb52e-9435-492b-b39d-f6ab737038bd-kube-api-access-jfjr9\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld\" (UID: \"30cfb52e-9435-492b-b39d-f6ab737038bd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld"
Jul 29 19:04:39.677575 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.677552    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh7qv\" (UniqueName: \"kubernetes.io/projected/185b4d06-9bb4-4f51-a4e3-6a156fc3fef2-kube-api-access-jh7qv\") pod \"volume-data-source-validator-549c6d54cf-p2dkv\" (UID: \"185b4d06-9bb4-4f51-a4e3-6a156fc3fef2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p2dkv"
Jul 29 19:04:39.677788 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.677766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxc92\" (UniqueName: \"kubernetes.io/projected/301cadcb-6dd5-4f64-b02c-bfaf77576609-kube-api-access-kxc92\") pod \"cluster-samples-operator-6748489656-gfsnv\" (UID: \"301cadcb-6dd5-4f64-b02c-bfaf77576609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"
Jul 29 19:04:39.769611 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.769540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bv5r\" (UniqueName: \"kubernetes.io/projected/a622c80f-34c3-4f10-97ca-e056df858797-kube-api-access-9bv5r\") pod \"console-operator-584dcb6b54-d52sf\" (UID: \"a622c80f-34c3-4f10-97ca-e056df858797\") " pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:39.769611 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.769589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:04:39.769786 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.769622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a622c80f-34c3-4f10-97ca-e056df858797-trusted-ca\") pod \"console-operator-584dcb6b54-d52sf\" (UID: \"a622c80f-34c3-4f10-97ca-e056df858797\") " pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:39.769786 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.769654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68123628-c31f-4b00-b091-0c1e8802972a-service-ca-bundle\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.769786 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.769674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a622c80f-34c3-4f10-97ca-e056df858797-serving-cert\") pod \"console-operator-584dcb6b54-d52sf\" (UID: \"a622c80f-34c3-4f10-97ca-e056df858797\") " pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:39.769786 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.769733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvv7j\" (UniqueName: \"kubernetes.io/projected/68123628-c31f-4b00-b091-0c1e8802972a-kube-api-access-rvv7j\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.769786 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.769761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:04:39.769786 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:39.769773    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 19:04:39.770050 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.769790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ffvj\" (UniqueName: \"kubernetes.io/projected/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-kube-api-access-2ffvj\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:04:39.770050 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:39.769833    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls podName:bf81bc82-dd40-4fbb-bc69-da2c1afa8517 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:40.269815488 +0000 UTC m=+246.569487969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vpxpc" (UID: "bf81bc82-dd40-4fbb-bc69-da2c1afa8517") : secret "cluster-monitoring-operator-tls" not found
Jul 29 19:04:39.770050 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.769978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68123628-c31f-4b00-b091-0c1e8802972a-serving-cert\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.770050 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.770014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68123628-c31f-4b00-b091-0c1e8802972a-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.770050 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.770045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a622c80f-34c3-4f10-97ca-e056df858797-config\") pod \"console-operator-584dcb6b54-d52sf\" (UID: \"a622c80f-34c3-4f10-97ca-e056df858797\") " pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:39.770318 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.770083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/68123628-c31f-4b00-b091-0c1e8802972a-snapshots\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.770318 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.770138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68123628-c31f-4b00-b091-0c1e8802972a-tmp\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.770423 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.770323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68123628-c31f-4b00-b091-0c1e8802972a-service-ca-bundle\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.770637 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.770614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:04:39.770885 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.770753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68123628-c31f-4b00-b091-0c1e8802972a-tmp\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.770992 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.770971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68123628-c31f-4b00-b091-0c1e8802972a-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.771057 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.771006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a622c80f-34c3-4f10-97ca-e056df858797-trusted-ca\") pod \"console-operator-584dcb6b54-d52sf\" (UID: \"a622c80f-34c3-4f10-97ca-e056df858797\") " pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:39.771057 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.771006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/68123628-c31f-4b00-b091-0c1e8802972a-snapshots\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.771120 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.771074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a622c80f-34c3-4f10-97ca-e056df858797-config\") pod \"console-operator-584dcb6b54-d52sf\" (UID: \"a622c80f-34c3-4f10-97ca-e056df858797\") " pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:39.772237 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.772215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a622c80f-34c3-4f10-97ca-e056df858797-serving-cert\") pod \"console-operator-584dcb6b54-d52sf\" (UID: \"a622c80f-34c3-4f10-97ca-e056df858797\") " pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:39.772390 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.772375    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68123628-c31f-4b00-b091-0c1e8802972a-serving-cert\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.775263 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.775247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p2dkv"
Jul 29 19:04:39.780054 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.780030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ffvj\" (UniqueName: \"kubernetes.io/projected/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-kube-api-access-2ffvj\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:04:39.780686 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.780666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvv7j\" (UniqueName: \"kubernetes.io/projected/68123628-c31f-4b00-b091-0c1e8802972a-kube-api-access-rvv7j\") pod \"insights-operator-5885fdb759-45jh7\" (UID: \"68123628-c31f-4b00-b091-0c1e8802972a\") " pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.781259 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.781244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bv5r\" (UniqueName: \"kubernetes.io/projected/a622c80f-34c3-4f10-97ca-e056df858797-kube-api-access-9bv5r\") pod \"console-operator-584dcb6b54-d52sf\" (UID: \"a622c80f-34c3-4f10-97ca-e056df858797\") " pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:39.781312 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.781303    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld"
Jul 29 19:04:39.793776 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.793750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569"
Jul 29 19:04:39.870694 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.870663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-45jh7"
Jul 29 19:04:39.886701 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.886674    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:39.919345 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.919276    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p2dkv"]
Jul 29 19:04:39.924277 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:04:39.924226    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185b4d06_9bb4_4f51_a4e3_6a156fc3fef2.slice/crio-3946cd8aaf263497945d29dbcf9921fa71348e00ef971c8abc1d2447eb212bae WatchSource:0}: Error finding container 3946cd8aaf263497945d29dbcf9921fa71348e00ef971c8abc1d2447eb212bae: Status 404 returned error can't find the container with id 3946cd8aaf263497945d29dbcf9921fa71348e00ef971c8abc1d2447eb212bae
Jul 29 19:04:39.937520 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.937171    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld"]
Jul 29 19:04:39.940180 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:04:39.940132    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cfb52e_9435_492b_b39d_f6ab737038bd.slice/crio-ae4676919c728216ce3f500f09c0845e680c090d5f245b0f779ee7073359d2fe WatchSource:0}: Error finding container ae4676919c728216ce3f500f09c0845e680c090d5f245b0f779ee7073359d2fe: Status 404 returned error can't find the container with id ae4676919c728216ce3f500f09c0845e680c090d5f245b0f779ee7073359d2fe
Jul 29 19:04:39.960391 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:39.960367    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569"]
Jul 29 19:04:39.961808 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:04:39.961779    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a486fe1_02df_4c80_8b23_83cb3e070a5a.slice/crio-dee9bb1b1f459df4b731c31e6210088bbe0822134cc43737f340f9279c1791d6 WatchSource:0}: Error finding container dee9bb1b1f459df4b731c31e6210088bbe0822134cc43737f340f9279c1791d6: Status 404 returned error can't find the container with id dee9bb1b1f459df4b731c31e6210088bbe0822134cc43737f340f9279c1791d6
Jul 29 19:04:40.021139 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:40.021114    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-45jh7"]
Jul 29 19:04:40.023112 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:04:40.023084    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68123628_c31f_4b00_b091_0c1e8802972a.slice/crio-dc5d38ff79a2be713d3f2f1ecc6e1089dd6ff021085a328a4ced9bd411b7b486 WatchSource:0}: Error finding container dc5d38ff79a2be713d3f2f1ecc6e1089dd6ff021085a328a4ced9bd411b7b486: Status 404 returned error can't find the container with id dc5d38ff79a2be713d3f2f1ecc6e1089dd6ff021085a328a4ced9bd411b7b486
Jul 29 19:04:40.032915 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:40.032888    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-d52sf"]
Jul 29 19:04:40.034880 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:04:40.034861    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda622c80f_34c3_4f10_97ca_e056df858797.slice/crio-a8e8d01af1c7be7b32f1003ef7e06e4146d1841c1d9ab6da7110b592cc656b64 WatchSource:0}: Error finding container a8e8d01af1c7be7b32f1003ef7e06e4146d1841c1d9ab6da7110b592cc656b64: Status 404 returned error can't find the container with id a8e8d01af1c7be7b32f1003ef7e06e4146d1841c1d9ab6da7110b592cc656b64
Jul 29 19:04:40.172729 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:40.172695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-gfsnv\" (UID: \"301cadcb-6dd5-4f64-b02c-bfaf77576609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"
Jul 29 19:04:40.172916 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:40.172874    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 19:04:40.172975 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:40.172950    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls podName:301cadcb-6dd5-4f64-b02c-bfaf77576609 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:41.172929738 +0000 UTC m=+247.472602216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls") pod "cluster-samples-operator-6748489656-gfsnv" (UID: "301cadcb-6dd5-4f64-b02c-bfaf77576609") : secret "samples-operator-tls" not found
Jul 29 19:04:40.274129 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:40.274054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:04:40.274289 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:40.274183    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 19:04:40.274289 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:40.274252    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls podName:bf81bc82-dd40-4fbb-bc69-da2c1afa8517 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:41.274238612 +0000 UTC m=+247.573911090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vpxpc" (UID: "bf81bc82-dd40-4fbb-bc69-da2c1afa8517") : secret "cluster-monitoring-operator-tls" not found
Jul 29 19:04:40.901357 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:40.901313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-d52sf" event={"ID":"a622c80f-34c3-4f10-97ca-e056df858797","Type":"ContainerStarted","Data":"a8e8d01af1c7be7b32f1003ef7e06e4146d1841c1d9ab6da7110b592cc656b64"}
Jul 29 19:04:40.902928 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:40.902899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-45jh7" event={"ID":"68123628-c31f-4b00-b091-0c1e8802972a","Type":"ContainerStarted","Data":"dc5d38ff79a2be713d3f2f1ecc6e1089dd6ff021085a328a4ced9bd411b7b486"}
Jul 29 19:04:40.904168 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:40.904139    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p2dkv" event={"ID":"185b4d06-9bb4-4f51-a4e3-6a156fc3fef2","Type":"ContainerStarted","Data":"3946cd8aaf263497945d29dbcf9921fa71348e00ef971c8abc1d2447eb212bae"}
Jul 29 19:04:40.905391 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:40.905364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569" event={"ID":"1a486fe1-02df-4c80-8b23-83cb3e070a5a","Type":"ContainerStarted","Data":"dee9bb1b1f459df4b731c31e6210088bbe0822134cc43737f340f9279c1791d6"}
Jul 29 19:04:40.907055 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:40.907035    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld" event={"ID":"30cfb52e-9435-492b-b39d-f6ab737038bd","Type":"ContainerStarted","Data":"ae4676919c728216ce3f500f09c0845e680c090d5f245b0f779ee7073359d2fe"}
Jul 29 19:04:41.181252 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:41.181138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-gfsnv\" (UID: \"301cadcb-6dd5-4f64-b02c-bfaf77576609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"
Jul 29 19:04:41.181407 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:41.181316    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 19:04:41.181407 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:41.181390    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls podName:301cadcb-6dd5-4f64-b02c-bfaf77576609 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:43.181369293 +0000 UTC m=+249.481041773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls") pod "cluster-samples-operator-6748489656-gfsnv" (UID: "301cadcb-6dd5-4f64-b02c-bfaf77576609") : secret "samples-operator-tls" not found
Jul 29 19:04:41.282616 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:41.282150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:04:41.282616 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:41.282301    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 19:04:41.282616 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:41.282369    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls podName:bf81bc82-dd40-4fbb-bc69-da2c1afa8517 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:43.282349132 +0000 UTC m=+249.582021644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vpxpc" (UID: "bf81bc82-dd40-4fbb-bc69-da2c1afa8517") : secret "cluster-monitoring-operator-tls" not found
Jul 29 19:04:41.912273 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:41.912232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p2dkv" event={"ID":"185b4d06-9bb4-4f51-a4e3-6a156fc3fef2","Type":"ContainerStarted","Data":"60ee0d1bf575ee5310761b8952156b7be98204bac732df0dc31768ab61500f48"}
Jul 29 19:04:41.930604 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:41.930546    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p2dkv" podStartSLOduration=1.4503532670000001 podStartE2EDuration="2.930524157s" totalImagesPullingTime="1.48017089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:04:41.929508923 +0000 UTC m=+248.229181423" watchObservedRunningTime="2026-07-29 19:04:41.930524157 +0000 UTC m=+248.230196670"
Jul 29 19:04:43.198728 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:43.198691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-gfsnv\" (UID: \"301cadcb-6dd5-4f64-b02c-bfaf77576609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"
Jul 29 19:04:43.199152 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:43.198869    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 19:04:43.199152 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:43.198951    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls podName:301cadcb-6dd5-4f64-b02c-bfaf77576609 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:47.198932788 +0000 UTC m=+253.498605272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls") pod "cluster-samples-operator-6748489656-gfsnv" (UID: "301cadcb-6dd5-4f64-b02c-bfaf77576609") : secret "samples-operator-tls" not found
Jul 29 19:04:43.299570 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:43.299531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:04:43.299746 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:43.299692    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 19:04:43.299800 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:43.299769    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls podName:bf81bc82-dd40-4fbb-bc69-da2c1afa8517 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:47.299750318 +0000 UTC m=+253.599422801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vpxpc" (UID: "bf81bc82-dd40-4fbb-bc69-da2c1afa8517") : secret "cluster-monitoring-operator-tls" not found
Jul 29 19:04:44.922901 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:44.922872    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/0.log"
Jul 29 19:04:44.923369 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:44.922911    2579 generic.go:356] "Generic (PLEG): container finished" podID="a622c80f-34c3-4f10-97ca-e056df858797" containerID="a1359665430adfc8e366d0acfd939395c690b3af95e751ec887a72d748ae5bf3" exitCode=255
Jul 29 19:04:44.923369 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:44.922983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-d52sf" event={"ID":"a622c80f-34c3-4f10-97ca-e056df858797","Type":"ContainerDied","Data":"a1359665430adfc8e366d0acfd939395c690b3af95e751ec887a72d748ae5bf3"}
Jul 29 19:04:44.923369 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:44.923219    2579 scope.go:117] "RemoveContainer" containerID="a1359665430adfc8e366d0acfd939395c690b3af95e751ec887a72d748ae5bf3"
Jul 29 19:04:44.924861 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:44.924830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-45jh7" event={"ID":"68123628-c31f-4b00-b091-0c1e8802972a","Type":"ContainerStarted","Data":"2b278102c93dc62701c08f558a4c8ee94649457a687075b98f473d871900cde6"}
Jul 29 19:04:44.926342 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:44.926312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569" event={"ID":"1a486fe1-02df-4c80-8b23-83cb3e070a5a","Type":"ContainerStarted","Data":"88d78abd6c17d03c3f1542a66be0aed67286defb46cebd6ec6e22ab2efcc7866"}
Jul 29 19:04:44.927735 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:44.927713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld" event={"ID":"30cfb52e-9435-492b-b39d-f6ab737038bd","Type":"ContainerStarted","Data":"f96b61398f76d7313b992ac3d135483ec83b9d309a7638d77c80cea83a39ca3d"}
Jul 29 19:04:44.954543 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:44.954493    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-45jh7" podStartSLOduration=1.991104223 podStartE2EDuration="5.954477373s" totalImagesPullingTime="3.96337315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:04:44.95349799 +0000 UTC m=+251.253170504" watchObservedRunningTime="2026-07-29 19:04:44.954477373 +0000 UTC m=+251.254149853"
Jul 29 19:04:44.968578 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:44.968538    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld" podStartSLOduration=1.9154736620000001 podStartE2EDuration="5.968525202s" totalImagesPullingTime="4.05305154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:04:44.967999291 +0000 UTC m=+251.267671790" watchObservedRunningTime="2026-07-29 19:04:44.968525202 +0000 UTC m=+251.268197704"
Jul 29 19:04:44.984598 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:44.984553    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569" podStartSLOduration=1.958042402 podStartE2EDuration="5.984534803s" totalImagesPullingTime="4.026492401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:04:44.984098784 +0000 UTC m=+251.283771282" watchObservedRunningTime="2026-07-29 19:04:44.984534803 +0000 UTC m=+251.284207304"
Jul 29 19:04:45.115253 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.114264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:04:45.115253 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:45.114706    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 19:04:45.115253 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:45.114785    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs podName:1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42 nodeName:}" failed. No retries permitted until 2026-07-29 19:06:47.1147522 +0000 UTC m=+373.414424694 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs") pod "network-metrics-daemon-zm2md" (UID: "1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42") : secret "metrics-daemon-secret" not found
Jul 29 19:04:45.369797 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.369711    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wqsgj"]
Jul 29 19:04:45.373318 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.373292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:45.375921 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.375889    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 19:04:45.376065 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.376033    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4lxqb\""
Jul 29 19:04:45.376125 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.376102    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 19:04:45.384802 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.384780    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wqsgj"]
Jul 29 19:04:45.518608 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.518572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-data-volume\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:45.518608 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.518607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:45.518797 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.518632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-crio-socket\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:45.518797 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.518661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb4ws\" (UniqueName: \"kubernetes.io/projected/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-api-access-pb4ws\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:45.518797 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.518739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:45.619768 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.619720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb4ws\" (UniqueName: \"kubernetes.io/projected/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-api-access-pb4ws\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:45.619980 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.619798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:45.619980 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.619905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-data-volume\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:45.619980 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.619929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:45.619980 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.619956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-crio-socket\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:45.620159 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:45.619928    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:04:45.620159 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.620036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-crio-socket\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:45.620159 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:45.620049    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:04:45.620159 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:45.620061    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm podName:acbc4c5f-87dc-474c-86a5-3d4551a92cc4 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:46.120039934 +0000 UTC m=+252.419712432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqsgj" (UID: "acbc4c5f-87dc-474c-86a5-3d4551a92cc4") : configmap "kube-rbac-proxy" not found
Jul 29 19:04:45.620159 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:45.620104    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls podName:acbc4c5f-87dc-474c-86a5-3d4551a92cc4 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:46.120090885 +0000 UTC m=+252.419763363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqsgj" (UID: "acbc4c5f-87dc-474c-86a5-3d4551a92cc4") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:04:45.620371 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.620302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-data-volume\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:45.630182 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.630152    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb4ws\" (UniqueName: \"kubernetes.io/projected/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-api-access-pb4ws\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:45.931781 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.931756    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:04:45.932207 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.932173    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/0.log"
Jul 29 19:04:45.932260 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.932218    2579 generic.go:356] "Generic (PLEG): container finished" podID="a622c80f-34c3-4f10-97ca-e056df858797" containerID="07543820923f31e300663dd7141703a529d8476bddbb55e4a3de0cdbbea498b5" exitCode=255
Jul 29 19:04:45.932302 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.932286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-d52sf" event={"ID":"a622c80f-34c3-4f10-97ca-e056df858797","Type":"ContainerDied","Data":"07543820923f31e300663dd7141703a529d8476bddbb55e4a3de0cdbbea498b5"}
Jul 29 19:04:45.932350 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.932323    2579 scope.go:117] "RemoveContainer" containerID="a1359665430adfc8e366d0acfd939395c690b3af95e751ec887a72d748ae5bf3"
Jul 29 19:04:45.932526 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:45.932507    2579 scope.go:117] "RemoveContainer" containerID="07543820923f31e300663dd7141703a529d8476bddbb55e4a3de0cdbbea498b5"
Jul 29 19:04:45.932714 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:45.932692    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-d52sf_openshift-console-operator(a622c80f-34c3-4f10-97ca-e056df858797)\"" pod="openshift-console-operator/console-operator-584dcb6b54-d52sf" podUID="a622c80f-34c3-4f10-97ca-e056df858797"
Jul 29 19:04:46.124713 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:46.124674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:46.124899 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:46.124756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:46.124899 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:46.124809    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:04:46.124899 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:46.124828    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:04:46.124899 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:46.124866    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls podName:acbc4c5f-87dc-474c-86a5-3d4551a92cc4 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:47.124851858 +0000 UTC m=+253.424524337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqsgj" (UID: "acbc4c5f-87dc-474c-86a5-3d4551a92cc4") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:04:46.124899 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:46.124878    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm podName:acbc4c5f-87dc-474c-86a5-3d4551a92cc4 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:47.124872733 +0000 UTC m=+253.424545211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqsgj" (UID: "acbc4c5f-87dc-474c-86a5-3d4551a92cc4") : configmap "kube-rbac-proxy" not found
Jul 29 19:04:46.936530 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:46.936500    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:04:46.936966 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:46.936862    2579 scope.go:117] "RemoveContainer" containerID="07543820923f31e300663dd7141703a529d8476bddbb55e4a3de0cdbbea498b5"
Jul 29 19:04:46.937018 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:46.937006    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-d52sf_openshift-console-operator(a622c80f-34c3-4f10-97ca-e056df858797)\"" pod="openshift-console-operator/console-operator-584dcb6b54-d52sf" podUID="a622c80f-34c3-4f10-97ca-e056df858797"
Jul 29 19:04:47.133538 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.133507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:47.133691 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.133594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:47.133691 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:47.133636    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:04:47.133691 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:47.133676    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:04:47.133806 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:47.133697    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm podName:acbc4c5f-87dc-474c-86a5-3d4551a92cc4 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:49.133682272 +0000 UTC m=+255.433354750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqsgj" (UID: "acbc4c5f-87dc-474c-86a5-3d4551a92cc4") : configmap "kube-rbac-proxy" not found
Jul 29 19:04:47.133806 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:47.133714    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls podName:acbc4c5f-87dc-474c-86a5-3d4551a92cc4 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:49.133703926 +0000 UTC m=+255.433376407 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqsgj" (UID: "acbc4c5f-87dc-474c-86a5-3d4551a92cc4") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:04:47.234677 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.234584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-gfsnv\" (UID: \"301cadcb-6dd5-4f64-b02c-bfaf77576609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"
Jul 29 19:04:47.234840 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:47.234719    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 19:04:47.234840 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:47.234779    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls podName:301cadcb-6dd5-4f64-b02c-bfaf77576609 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:55.234764153 +0000 UTC m=+261.534436631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls") pod "cluster-samples-operator-6748489656-gfsnv" (UID: "301cadcb-6dd5-4f64-b02c-bfaf77576609") : secret "samples-operator-tls" not found
Jul 29 19:04:47.335915 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.335870    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:04:47.336095 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:47.336035    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 19:04:47.336134 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:47.336103    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls podName:bf81bc82-dd40-4fbb-bc69-da2c1afa8517 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:55.336085983 +0000 UTC m=+261.635758461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vpxpc" (UID: "bf81bc82-dd40-4fbb-bc69-da2c1afa8517") : secret "cluster-monitoring-operator-tls" not found
Jul 29 19:04:47.535778 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.535706    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l9btm_fab1c53d-2ed6-48b8-86b6-6244542b8750/dns-node-resolver/0.log"
Jul 29 19:04:47.568363 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.568329    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-9p52p"]
Jul 29 19:04:47.572559 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.572542    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-9p52p"
Jul 29 19:04:47.575364 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.575338    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 29 19:04:47.575481 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.575364    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9vfv6\""
Jul 29 19:04:47.575481 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.575340    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 29 19:04:47.575481 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.575467    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 29 19:04:47.576560 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.576545    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 29 19:04:47.579966 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.579943    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-9p52p"]
Jul 29 19:04:47.739359 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.739327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f2s6\" (UniqueName: \"kubernetes.io/projected/1f85621a-4e2d-4249-998e-5eed6b3030d0-kube-api-access-6f2s6\") pod \"service-ca-658d7b7c78-9p52p\" (UID: \"1f85621a-4e2d-4249-998e-5eed6b3030d0\") " pod="openshift-service-ca/service-ca-658d7b7c78-9p52p"
Jul 29 19:04:47.739525 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.739450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f85621a-4e2d-4249-998e-5eed6b3030d0-signing-cabundle\") pod \"service-ca-658d7b7c78-9p52p\" (UID: \"1f85621a-4e2d-4249-998e-5eed6b3030d0\") " pod="openshift-service-ca/service-ca-658d7b7c78-9p52p"
Jul 29 19:04:47.739525 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.739511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1f85621a-4e2d-4249-998e-5eed6b3030d0-signing-key\") pod \"service-ca-658d7b7c78-9p52p\" (UID: \"1f85621a-4e2d-4249-998e-5eed6b3030d0\") " pod="openshift-service-ca/service-ca-658d7b7c78-9p52p"
Jul 29 19:04:47.840531 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.840453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1f85621a-4e2d-4249-998e-5eed6b3030d0-signing-key\") pod \"service-ca-658d7b7c78-9p52p\" (UID: \"1f85621a-4e2d-4249-998e-5eed6b3030d0\") " pod="openshift-service-ca/service-ca-658d7b7c78-9p52p"
Jul 29 19:04:47.840531 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.840507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f2s6\" (UniqueName: \"kubernetes.io/projected/1f85621a-4e2d-4249-998e-5eed6b3030d0-kube-api-access-6f2s6\") pod \"service-ca-658d7b7c78-9p52p\" (UID: \"1f85621a-4e2d-4249-998e-5eed6b3030d0\") " pod="openshift-service-ca/service-ca-658d7b7c78-9p52p"
Jul 29 19:04:47.840703 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.840685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f85621a-4e2d-4249-998e-5eed6b3030d0-signing-cabundle\") pod \"service-ca-658d7b7c78-9p52p\" (UID: \"1f85621a-4e2d-4249-998e-5eed6b3030d0\") " pod="openshift-service-ca/service-ca-658d7b7c78-9p52p"
Jul 29 19:04:47.841300 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.841277    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f85621a-4e2d-4249-998e-5eed6b3030d0-signing-cabundle\") pod \"service-ca-658d7b7c78-9p52p\" (UID: \"1f85621a-4e2d-4249-998e-5eed6b3030d0\") " pod="openshift-service-ca/service-ca-658d7b7c78-9p52p"
Jul 29 19:04:47.842973 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.842952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1f85621a-4e2d-4249-998e-5eed6b3030d0-signing-key\") pod \"service-ca-658d7b7c78-9p52p\" (UID: \"1f85621a-4e2d-4249-998e-5eed6b3030d0\") " pod="openshift-service-ca/service-ca-658d7b7c78-9p52p"
Jul 29 19:04:47.849278 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.849257    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f2s6\" (UniqueName: \"kubernetes.io/projected/1f85621a-4e2d-4249-998e-5eed6b3030d0-kube-api-access-6f2s6\") pod \"service-ca-658d7b7c78-9p52p\" (UID: \"1f85621a-4e2d-4249-998e-5eed6b3030d0\") " pod="openshift-service-ca/service-ca-658d7b7c78-9p52p"
Jul 29 19:04:47.882109 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:47.882073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-9p52p"
Jul 29 19:04:48.008350 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:48.008125    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-9p52p"]
Jul 29 19:04:48.010947 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:04:48.010908    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f85621a_4e2d_4249_998e_5eed6b3030d0.slice/crio-c3794669527393e7a82154958a40f51c8b8db65f737ff4b35471bc3b927d0fe4 WatchSource:0}: Error finding container c3794669527393e7a82154958a40f51c8b8db65f737ff4b35471bc3b927d0fe4: Status 404 returned error can't find the container with id c3794669527393e7a82154958a40f51c8b8db65f737ff4b35471bc3b927d0fe4
Jul 29 19:04:48.136658 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:48.136633    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7tb67_841e2b86-5a6c-43d7-9919-da697a9ce802/node-ca/0.log"
Jul 29 19:04:48.952306 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:48.952270    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-9p52p" event={"ID":"1f85621a-4e2d-4249-998e-5eed6b3030d0","Type":"ContainerStarted","Data":"66b54bfe71b474730917f3462c3914914d66d6f9ab47f5979a7312ec856818d2"}
Jul 29 19:04:48.952461 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:48.952325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-9p52p" event={"ID":"1f85621a-4e2d-4249-998e-5eed6b3030d0","Type":"ContainerStarted","Data":"c3794669527393e7a82154958a40f51c8b8db65f737ff4b35471bc3b927d0fe4"}
Jul 29 19:04:48.968949 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:48.968898    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-9p52p" podStartSLOduration=1.968881396 podStartE2EDuration="1.968881396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:04:48.96792831 +0000 UTC m=+255.267600810" watchObservedRunningTime="2026-07-29 19:04:48.968881396 +0000 UTC m=+255.268553942"
Jul 29 19:04:49.137538 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:49.137504    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qkzwn_614641fc-597a-4b94-ba7c-33f078d5ed1c/serve-healthcheck-canary/0.log"
Jul 29 19:04:49.150843 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:49.150806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:49.150976 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:49.150920    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:04:49.150976 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:49.150950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:49.151066 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:49.150982    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm podName:acbc4c5f-87dc-474c-86a5-3d4551a92cc4 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:53.150965952 +0000 UTC m=+259.450638430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqsgj" (UID: "acbc4c5f-87dc-474c-86a5-3d4551a92cc4") : configmap "kube-rbac-proxy" not found
Jul 29 19:04:49.151066 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:49.151051    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:04:49.151138 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:49.151106    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls podName:acbc4c5f-87dc-474c-86a5-3d4551a92cc4 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:53.151091744 +0000 UTC m=+259.450764229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqsgj" (UID: "acbc4c5f-87dc-474c-86a5-3d4551a92cc4") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:04:49.887009 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:49.886963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:49.887171 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:49.887033    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:04:49.887378 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:49.887366    2579 scope.go:117] "RemoveContainer" containerID="07543820923f31e300663dd7141703a529d8476bddbb55e4a3de0cdbbea498b5"
Jul 29 19:04:49.887525 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:49.887508    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-d52sf_openshift-console-operator(a622c80f-34c3-4f10-97ca-e056df858797)\"" pod="openshift-console-operator/console-operator-584dcb6b54-d52sf" podUID="a622c80f-34c3-4f10-97ca-e056df858797"
Jul 29 19:04:53.181739 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:53.181701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:53.182284 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:53.181842    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:04:53.182284 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:53.181898    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm podName:acbc4c5f-87dc-474c-86a5-3d4551a92cc4 nodeName:}" failed. No retries permitted until 2026-07-29 19:05:01.181883815 +0000 UTC m=+267.481556293 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqsgj" (UID: "acbc4c5f-87dc-474c-86a5-3d4551a92cc4") : configmap "kube-rbac-proxy" not found
Jul 29 19:04:53.182284 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:53.181987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:04:53.182284 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:53.182079    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:04:53.182284 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:53.182109    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls podName:acbc4c5f-87dc-474c-86a5-3d4551a92cc4 nodeName:}" failed. No retries permitted until 2026-07-29 19:05:01.182102001 +0000 UTC m=+267.481774479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqsgj" (UID: "acbc4c5f-87dc-474c-86a5-3d4551a92cc4") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:04:55.299786 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:55.299685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-gfsnv\" (UID: \"301cadcb-6dd5-4f64-b02c-bfaf77576609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"
Jul 29 19:04:55.302487 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:55.302460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301cadcb-6dd5-4f64-b02c-bfaf77576609-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-gfsnv\" (UID: \"301cadcb-6dd5-4f64-b02c-bfaf77576609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"
Jul 29 19:04:55.388754 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:55.388713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"
Jul 29 19:04:55.401106 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:55.401066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:04:55.401343 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:55.401317    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 19:04:55.401430 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:04:55.401415    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls podName:bf81bc82-dd40-4fbb-bc69-da2c1afa8517 nodeName:}" failed. No retries permitted until 2026-07-29 19:05:11.401389484 +0000 UTC m=+277.701062016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vpxpc" (UID: "bf81bc82-dd40-4fbb-bc69-da2c1afa8517") : secret "cluster-monitoring-operator-tls" not found
Jul 29 19:04:55.520983 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:55.520950    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv"]
Jul 29 19:04:55.972754 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:55.972721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv" event={"ID":"301cadcb-6dd5-4f64-b02c-bfaf77576609","Type":"ContainerStarted","Data":"99a7842988ca84c8162caba38dc6a9f386f7758ebca66a746eb51fe051f9dc9c"}
Jul 29 19:04:57.983135 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:57.983079    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv" event={"ID":"301cadcb-6dd5-4f64-b02c-bfaf77576609","Type":"ContainerStarted","Data":"2e5451b8b6b0122d94b072b5275ca7bfd03a643456026d7d0ab4ffda506d4d7b"}
Jul 29 19:04:57.983511 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:57.983172    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv" event={"ID":"301cadcb-6dd5-4f64-b02c-bfaf77576609","Type":"ContainerStarted","Data":"fac5568d4aa108267c3a2798d4c721135337ed59029b57adca6b79e08accdfaf"}
Jul 29 19:04:59.002358 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:04:59.002296    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gfsnv" podStartSLOduration=17.744419448 podStartE2EDuration="20.002277934s" totalImagesPullingTime="2.257858486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:04:59.001130179 +0000 UTC m=+265.300802701" watchObservedRunningTime="2026-07-29 19:04:59.002277934 +0000 UTC m=+265.301950433"
Jul 29 19:05:01.259975 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:01.259929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:05:01.260416 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:01.259998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:05:01.260553 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:01.260535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:05:01.262701 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:01.262675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acbc4c5f-87dc-474c-86a5-3d4551a92cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqsgj\" (UID: \"acbc4c5f-87dc-474c-86a5-3d4551a92cc4\") " pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:05:01.283390 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:01.283362    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqsgj"
Jul 29 19:05:01.443672 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:01.443495    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wqsgj"]
Jul 29 19:05:01.445936 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:05:01.445903    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbc4c5f_87dc_474c_86a5_3d4551a92cc4.slice/crio-e7d82a8ff0988035c585ddfeb97a739322d15eed787068c7d9a92038bcbd5e48 WatchSource:0}: Error finding container e7d82a8ff0988035c585ddfeb97a739322d15eed787068c7d9a92038bcbd5e48: Status 404 returned error can't find the container with id e7d82a8ff0988035c585ddfeb97a739322d15eed787068c7d9a92038bcbd5e48
Jul 29 19:05:01.996465 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:01.996423    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqsgj" event={"ID":"acbc4c5f-87dc-474c-86a5-3d4551a92cc4","Type":"ContainerStarted","Data":"e7d82a8ff0988035c585ddfeb97a739322d15eed787068c7d9a92038bcbd5e48"}
Jul 29 19:05:05.191042 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:05.191004    2579 scope.go:117] "RemoveContainer" containerID="07543820923f31e300663dd7141703a529d8476bddbb55e4a3de0cdbbea498b5"
Jul 29 19:05:07.016273 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:07.016246    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:05:07.016768 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:07.016336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-d52sf" event={"ID":"a622c80f-34c3-4f10-97ca-e056df858797","Type":"ContainerStarted","Data":"d4c28ad419bc6e5488b0d1b5fceef6d62b5d9f3ccf3469e7f106e5aa3bc3bdaf"}
Jul 29 19:05:07.016832 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:07.016779    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:05:07.018170 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:07.018145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqsgj" event={"ID":"acbc4c5f-87dc-474c-86a5-3d4551a92cc4","Type":"ContainerStarted","Data":"39981b95dea57de65918fd5ed5746973702c01256e02c37d46d493ab3f865b1d"}
Jul 29 19:05:07.021865 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:07.021844    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-d52sf"
Jul 29 19:05:07.033669 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:07.033623    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-d52sf" podStartSLOduration=24.076775794 podStartE2EDuration="28.033609413s" totalImagesPullingTime="3.956833619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:07.032730933 +0000 UTC m=+273.332403455" watchObservedRunningTime="2026-07-29 19:05:07.033609413 +0000 UTC m=+273.333281954"
Jul 29 19:05:10.143242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:10.143209    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-lvx6t"]
Jul 29 19:05:10.158546 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:10.158405    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b46dcb55c-q4dhq"]
Jul 29 19:05:10.158546 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:10.158469    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-lvx6t"]
Jul 29 19:05:10.158800 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:10.158647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-lvx6t"
Jul 29 19:05:10.158800 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:05:10.158716    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq" podUID="d6fcf51f-c473-441e-8136-e5c300f7dc65"
Jul 29 19:05:10.161445 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:10.161384    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 19:05:10.161445 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:10.161430    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-pcfj6\""
Jul 29 19:05:10.161665 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:10.161501    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 19:05:10.238449 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:10.238406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9lgn\" (UniqueName: \"kubernetes.io/projected/47785f11-d0a3-4108-a84c-71da470dca6c-kube-api-access-f9lgn\") pod \"downloads-7c97dc4555-lvx6t\" (UID: \"47785f11-d0a3-4108-a84c-71da470dca6c\") " pod="openshift-console/downloads-7c97dc4555-lvx6t"
Jul 29 19:05:10.339616 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:10.339570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9lgn\" (UniqueName: \"kubernetes.io/projected/47785f11-d0a3-4108-a84c-71da470dca6c-kube-api-access-f9lgn\") pod \"downloads-7c97dc4555-lvx6t\" (UID: \"47785f11-d0a3-4108-a84c-71da470dca6c\") " pod="openshift-console/downloads-7c97dc4555-lvx6t"
Jul 29 19:05:10.349158 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:10.349122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9lgn\" (UniqueName: \"kubernetes.io/projected/47785f11-d0a3-4108-a84c-71da470dca6c-kube-api-access-f9lgn\") pod \"downloads-7c97dc4555-lvx6t\" (UID: \"47785f11-d0a3-4108-a84c-71da470dca6c\") " pod="openshift-console/downloads-7c97dc4555-lvx6t"
Jul 29 19:05:10.471459 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:10.471381    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-lvx6t"
Jul 29 19:05:11.031519 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.031478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:05:11.049672 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.049641    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:05:11.148291 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.148250    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-certificates\") pod \"d6fcf51f-c473-441e-8136-e5c300f7dc65\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") "
Jul 29 19:05:11.148780 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.148310    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf51f-c473-441e-8136-e5c300f7dc65-ca-trust-extracted\") pod \"d6fcf51f-c473-441e-8136-e5c300f7dc65\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") "
Jul 29 19:05:11.148780 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.148363    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6fcf51f-c473-441e-8136-e5c300f7dc65-installation-pull-secrets\") pod \"d6fcf51f-c473-441e-8136-e5c300f7dc65\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") "
Jul 29 19:05:11.148780 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.148398    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6fcf51f-c473-441e-8136-e5c300f7dc65-image-registry-private-configuration\") pod \"d6fcf51f-c473-441e-8136-e5c300f7dc65\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") "
Jul 29 19:05:11.148780 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.148459    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-bound-sa-token\") pod \"d6fcf51f-c473-441e-8136-e5c300f7dc65\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") "
Jul 29 19:05:11.148780 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.148486    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6fcf51f-c473-441e-8136-e5c300f7dc65-trusted-ca\") pod \"d6fcf51f-c473-441e-8136-e5c300f7dc65\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") "
Jul 29 19:05:11.148780 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.148521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-697hr\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-kube-api-access-697hr\") pod \"d6fcf51f-c473-441e-8136-e5c300f7dc65\" (UID: \"d6fcf51f-c473-441e-8136-e5c300f7dc65\") "
Jul 29 19:05:11.148780 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.148712    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d6fcf51f-c473-441e-8136-e5c300f7dc65" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:05:11.148780 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.148724    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fcf51f-c473-441e-8136-e5c300f7dc65-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d6fcf51f-c473-441e-8136-e5c300f7dc65" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:05:11.149173 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.148912    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-certificates\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:11.149173 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.148934    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf51f-c473-441e-8136-e5c300f7dc65-ca-trust-extracted\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:11.149173 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.149082    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6fcf51f-c473-441e-8136-e5c300f7dc65-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d6fcf51f-c473-441e-8136-e5c300f7dc65" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:05:11.151504 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.151476    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6fcf51f-c473-441e-8136-e5c300f7dc65-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d6fcf51f-c473-441e-8136-e5c300f7dc65" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:05:11.151504 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.151488    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-kube-api-access-697hr" (OuterVolumeSpecName: "kube-api-access-697hr") pod "d6fcf51f-c473-441e-8136-e5c300f7dc65" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65"). InnerVolumeSpecName "kube-api-access-697hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:05:11.151655 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.151513    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d6fcf51f-c473-441e-8136-e5c300f7dc65" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:05:11.151739 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.151718    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6fcf51f-c473-441e-8136-e5c300f7dc65-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d6fcf51f-c473-441e-8136-e5c300f7dc65" (UID: "d6fcf51f-c473-441e-8136-e5c300f7dc65"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:05:11.249526 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.249488    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6fcf51f-c473-441e-8136-e5c300f7dc65-installation-pull-secrets\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:11.249526 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.249527    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6fcf51f-c473-441e-8136-e5c300f7dc65-image-registry-private-configuration\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:11.249723 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.249544    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-bound-sa-token\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:11.249723 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.249558    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6fcf51f-c473-441e-8136-e5c300f7dc65-trusted-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:11.249723 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.249571    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-697hr\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-kube-api-access-697hr\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:11.325631 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.325610    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-lvx6t"]
Jul 29 19:05:11.327669 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:05:11.327634    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47785f11_d0a3_4108_a84c_71da470dca6c.slice/crio-db530ca025badd5d5a05a105d0f34a78ca7f1db671ae8d59063472841de85be8 WatchSource:0}: Error finding container db530ca025badd5d5a05a105d0f34a78ca7f1db671ae8d59063472841de85be8: Status 404 returned error can't find the container with id db530ca025badd5d5a05a105d0f34a78ca7f1db671ae8d59063472841de85be8
Jul 29 19:05:11.451073 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.451033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:05:11.453861 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.453835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf81bc82-dd40-4fbb-bc69-da2c1afa8517-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vpxpc\" (UID: \"bf81bc82-dd40-4fbb-bc69-da2c1afa8517\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:05:11.683464 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.683424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"
Jul 29 19:05:11.813401 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:11.813378    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc"]
Jul 29 19:05:11.815593 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:05:11.815552    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf81bc82_dd40_4fbb_bc69_da2c1afa8517.slice/crio-309fdd3b433fb7173730673c0b2c538a127ce5333ebb30590f7416cc2e25c9b8 WatchSource:0}: Error finding container 309fdd3b433fb7173730673c0b2c538a127ce5333ebb30590f7416cc2e25c9b8: Status 404 returned error can't find the container with id 309fdd3b433fb7173730673c0b2c538a127ce5333ebb30590f7416cc2e25c9b8
Jul 29 19:05:12.039363 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:12.039187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqsgj" event={"ID":"acbc4c5f-87dc-474c-86a5-3d4551a92cc4","Type":"ContainerStarted","Data":"8038fc73d8464025ca0477fad891020343f0a2b092074eeb38083e9ff168852c"}
Jul 29 19:05:12.040870 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:12.040840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc" event={"ID":"bf81bc82-dd40-4fbb-bc69-da2c1afa8517","Type":"ContainerStarted","Data":"309fdd3b433fb7173730673c0b2c538a127ce5333ebb30590f7416cc2e25c9b8"}
Jul 29 19:05:12.043535 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:12.042140    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b46dcb55c-q4dhq"
Jul 29 19:05:12.043535 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:12.042667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-lvx6t" event={"ID":"47785f11-d0a3-4108-a84c-71da470dca6c","Type":"ContainerStarted","Data":"db530ca025badd5d5a05a105d0f34a78ca7f1db671ae8d59063472841de85be8"}
Jul 29 19:05:12.079394 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:12.079325    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b46dcb55c-q4dhq"]
Jul 29 19:05:12.084140 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:12.084109    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b46dcb55c-q4dhq"]
Jul 29 19:05:12.157680 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:12.157644    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6fcf51f-c473-441e-8136-e5c300f7dc65-registry-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:12.196527 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:12.196265    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6fcf51f-c473-441e-8136-e5c300f7dc65" path="/var/lib/kubelet/pods/d6fcf51f-c473-441e-8136-e5c300f7dc65/volumes"
Jul 29 19:05:15.270109 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.270055    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cf47c5b58-l6zdl"]
Jul 29 19:05:15.294429 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.294390    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf47c5b58-l6zdl"]
Jul 29 19:05:15.294629 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.294527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.297842 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.297803    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 19:05:15.297842 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.297803    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 19:05:15.298069 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.297903    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 19:05:15.298069 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.297938    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 19:05:15.298173 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.298100    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 19:05:15.299326 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.299301    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-plq5m\""
Jul 29 19:05:15.389952 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.389915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhklv\" (UniqueName: \"kubernetes.io/projected/db5da208-6fa5-4048-9f63-974e985d6074-kube-api-access-xhklv\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.390152 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.389979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db5da208-6fa5-4048-9f63-974e985d6074-console-serving-cert\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.390152 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.390067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-service-ca\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.390152 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.390100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-console-config\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.390152 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.390134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-oauth-serving-cert\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.390449 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.390166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db5da208-6fa5-4048-9f63-974e985d6074-console-oauth-config\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.490728 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.490680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhklv\" (UniqueName: \"kubernetes.io/projected/db5da208-6fa5-4048-9f63-974e985d6074-kube-api-access-xhklv\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.490948 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.490765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db5da208-6fa5-4048-9f63-974e985d6074-console-serving-cert\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.490948 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.490829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-service-ca\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.490948 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.490853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-console-config\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.490948 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.490883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-oauth-serving-cert\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.490948 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.490905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db5da208-6fa5-4048-9f63-974e985d6074-console-oauth-config\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.491771 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.491742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-console-config\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.491964 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.491938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-service-ca\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.492335 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.492308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-oauth-serving-cert\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.493973 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.493944    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db5da208-6fa5-4048-9f63-974e985d6074-console-serving-cert\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.494092 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.494007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db5da208-6fa5-4048-9f63-974e985d6074-console-oauth-config\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.501550 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.501523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhklv\" (UniqueName: \"kubernetes.io/projected/db5da208-6fa5-4048-9f63-974e985d6074-kube-api-access-xhklv\") pod \"console-6cf47c5b58-l6zdl\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") " pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.606558 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.606457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:15.697289 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:05:15.697238    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-79x22" podUID="15228df5-30d8-4cd8-8de4-a55d961a28d8"
Jul 29 19:05:15.870487 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:15.870453    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf47c5b58-l6zdl"]
Jul 29 19:05:15.872594 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:05:15.872552    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5da208_6fa5_4048_9f63_974e985d6074.slice/crio-fc58b01e18c45328578ad63a2d96659e1ae4583a64fa2951183fe2efb644efca WatchSource:0}: Error finding container fc58b01e18c45328578ad63a2d96659e1ae4583a64fa2951183fe2efb644efca: Status 404 returned error can't find the container with id fc58b01e18c45328578ad63a2d96659e1ae4583a64fa2951183fe2efb644efca
Jul 29 19:05:16.061082 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.061023    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf47c5b58-l6zdl" event={"ID":"db5da208-6fa5-4048-9f63-974e985d6074","Type":"ContainerStarted","Data":"fc58b01e18c45328578ad63a2d96659e1ae4583a64fa2951183fe2efb644efca"}
Jul 29 19:05:16.063759 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.063724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqsgj" event={"ID":"acbc4c5f-87dc-474c-86a5-3d4551a92cc4","Type":"ContainerStarted","Data":"655cdc88ea5306a3c06b41e5a794b332258642b84b998669e6fa3445819a8c17"}
Jul 29 19:05:16.066416 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.066379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc" event={"ID":"bf81bc82-dd40-4fbb-bc69-da2c1afa8517","Type":"ContainerStarted","Data":"8f7066a87ea49d0b8b8c6bf71c140ae03c68a161edc286205cf4ae8daafd69a5"}
Jul 29 19:05:16.066416 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.066404    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-79x22"
Jul 29 19:05:16.084121 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.084067    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wqsgj" podStartSLOduration=16.920978229 podStartE2EDuration="31.084047127s" totalImagesPullingTime="14.163068898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:04:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:16.082393252 +0000 UTC m=+282.382065771" watchObservedRunningTime="2026-07-29 19:05:16.084047127 +0000 UTC m=+282.383719605"
Jul 29 19:05:16.232660 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.232595    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vpxpc" podStartSLOduration=33.330614046 podStartE2EDuration="37.232573487s" totalImagesPullingTime="3.901959441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:16.098576897 +0000 UTC m=+282.398249398" watchObservedRunningTime="2026-07-29 19:05:16.232573487 +0000 UTC m=+282.532245986"
Jul 29 19:05:16.233288 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.233268    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-msz52"]
Jul 29 19:05:16.239901 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.239868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-msz52"
Jul 29 19:05:16.242788 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.242696    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bgmf4\""
Jul 29 19:05:16.242949 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.242884    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-msz52"]
Jul 29 19:05:16.242949 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.242946    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 29 19:05:16.300802 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.300759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/24a344c4-d14b-4181-952f-0c6aef93cb56-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-msz52\" (UID: \"24a344c4-d14b-4181-952f-0c6aef93cb56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-msz52"
Jul 29 19:05:16.401521 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.401485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/24a344c4-d14b-4181-952f-0c6aef93cb56-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-msz52\" (UID: \"24a344c4-d14b-4181-952f-0c6aef93cb56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-msz52"
Jul 29 19:05:16.404406 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.404370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/24a344c4-d14b-4181-952f-0c6aef93cb56-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-msz52\" (UID: \"24a344c4-d14b-4181-952f-0c6aef93cb56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-msz52"
Jul 29 19:05:16.551512 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.551404    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-msz52"
Jul 29 19:05:16.697990 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:16.697959    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-msz52"]
Jul 29 19:05:16.700987 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:05:16.700950    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a344c4_d14b_4181_952f_0c6aef93cb56.slice/crio-6866d2bc327cbe3b414500a378dcb9f1419d8ecc63ac403df101a31b5bfceb07 WatchSource:0}: Error finding container 6866d2bc327cbe3b414500a378dcb9f1419d8ecc63ac403df101a31b5bfceb07: Status 404 returned error can't find the container with id 6866d2bc327cbe3b414500a378dcb9f1419d8ecc63ac403df101a31b5bfceb07
Jul 29 19:05:17.071724 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:17.071676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-msz52" event={"ID":"24a344c4-d14b-4181-952f-0c6aef93cb56","Type":"ContainerStarted","Data":"6866d2bc327cbe3b414500a378dcb9f1419d8ecc63ac403df101a31b5bfceb07"}
Jul 29 19:05:19.083010 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.082966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-msz52" event={"ID":"24a344c4-d14b-4181-952f-0c6aef93cb56","Type":"ContainerStarted","Data":"7f447d85bdd967802b2719e6d9c8cfeb2d53a0dcba3b2b40391cc6cee50f286f"}
Jul 29 19:05:19.083603 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.083553    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-msz52"
Jul 29 19:05:19.089092 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.089033    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-msz52"
Jul 29 19:05:19.099339 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.099280    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-msz52" podStartSLOduration=1.790654379 podStartE2EDuration="3.09925768s" totalImagesPullingTime="1.308603301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:19.098369913 +0000 UTC m=+285.398042413" watchObservedRunningTime="2026-07-29 19:05:19.09925768 +0000 UTC m=+285.398930180"
Jul 29 19:05:19.290364 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.290321    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"]
Jul 29 19:05:19.296087 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.296053    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"
Jul 29 19:05:19.299400 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.299165    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 29 19:05:19.299400 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.299176    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wxdzq\""
Jul 29 19:05:19.299928 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.299905    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 29 19:05:19.300303 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.300131    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 19:05:19.303810 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.303770    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"]
Jul 29 19:05:19.432210 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.432167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a07e4ea-2748-4ca4-b93c-ba5a5652f012-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-mlnw2\" (UID: \"4a07e4ea-2748-4ca4-b93c-ba5a5652f012\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"
Jul 29 19:05:19.432418 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.432341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a07e4ea-2748-4ca4-b93c-ba5a5652f012-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-mlnw2\" (UID: \"4a07e4ea-2748-4ca4-b93c-ba5a5652f012\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"
Jul 29 19:05:19.432418 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.432406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k67x8\" (UniqueName: \"kubernetes.io/projected/4a07e4ea-2748-4ca4-b93c-ba5a5652f012-kube-api-access-k67x8\") pod \"prometheus-operator-84d69f794d-mlnw2\" (UID: \"4a07e4ea-2748-4ca4-b93c-ba5a5652f012\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"
Jul 29 19:05:19.432528 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.432468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a07e4ea-2748-4ca4-b93c-ba5a5652f012-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-mlnw2\" (UID: \"4a07e4ea-2748-4ca4-b93c-ba5a5652f012\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"
Jul 29 19:05:19.533082 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.533034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a07e4ea-2748-4ca4-b93c-ba5a5652f012-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-mlnw2\" (UID: \"4a07e4ea-2748-4ca4-b93c-ba5a5652f012\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"
Jul 29 19:05:19.533337 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.533148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a07e4ea-2748-4ca4-b93c-ba5a5652f012-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-mlnw2\" (UID: \"4a07e4ea-2748-4ca4-b93c-ba5a5652f012\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"
Jul 29 19:05:19.533337 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.533209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:05:19.533337 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.533239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k67x8\" (UniqueName: \"kubernetes.io/projected/4a07e4ea-2748-4ca4-b93c-ba5a5652f012-kube-api-access-k67x8\") pod \"prometheus-operator-84d69f794d-mlnw2\" (UID: \"4a07e4ea-2748-4ca4-b93c-ba5a5652f012\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"
Jul 29 19:05:19.533337 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.533280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a07e4ea-2748-4ca4-b93c-ba5a5652f012-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-mlnw2\" (UID: \"4a07e4ea-2748-4ca4-b93c-ba5a5652f012\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"
Jul 29 19:05:19.534216 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.534153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a07e4ea-2748-4ca4-b93c-ba5a5652f012-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-mlnw2\" (UID: \"4a07e4ea-2748-4ca4-b93c-ba5a5652f012\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"
Jul 29 19:05:19.536285 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.536237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15228df5-30d8-4cd8-8de4-a55d961a28d8-metrics-tls\") pod \"dns-default-79x22\" (UID: \"15228df5-30d8-4cd8-8de4-a55d961a28d8\") " pod="openshift-dns/dns-default-79x22"
Jul 29 19:05:19.536401 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.536348    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a07e4ea-2748-4ca4-b93c-ba5a5652f012-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-mlnw2\" (UID: \"4a07e4ea-2748-4ca4-b93c-ba5a5652f012\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"
Jul 29 19:05:19.536745 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.536718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a07e4ea-2748-4ca4-b93c-ba5a5652f012-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-mlnw2\" (UID: \"4a07e4ea-2748-4ca4-b93c-ba5a5652f012\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"
Jul 29 19:05:19.542467 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.542432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k67x8\" (UniqueName: \"kubernetes.io/projected/4a07e4ea-2748-4ca4-b93c-ba5a5652f012-kube-api-access-k67x8\") pod \"prometheus-operator-84d69f794d-mlnw2\" (UID: \"4a07e4ea-2748-4ca4-b93c-ba5a5652f012\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"
Jul 29 19:05:19.609310 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.609274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"
Jul 29 19:05:19.670321 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.670289    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7dnhc\""
Jul 29 19:05:19.677796 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.677763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-79x22"
Jul 29 19:05:19.910545 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.907895    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-mlnw2"]
Jul 29 19:05:19.929518 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:19.929483    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-79x22"]
Jul 29 19:05:19.933000 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:05:19.932971    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15228df5_30d8_4cd8_8de4_a55d961a28d8.slice/crio-0c7c7833b08ab40337338ed5d5d245ad3e89a87473285029d562e19a7b765985 WatchSource:0}: Error finding container 0c7c7833b08ab40337338ed5d5d245ad3e89a87473285029d562e19a7b765985: Status 404 returned error can't find the container with id 0c7c7833b08ab40337338ed5d5d245ad3e89a87473285029d562e19a7b765985
Jul 29 19:05:20.087837 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:20.087721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2" event={"ID":"4a07e4ea-2748-4ca4-b93c-ba5a5652f012","Type":"ContainerStarted","Data":"92034f4651ed33ca6371eb2d8d56c5cfed6ca03b49128007855043da38952b1e"}
Jul 29 19:05:20.088903 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:20.088868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-79x22" event={"ID":"15228df5-30d8-4cd8-8de4-a55d961a28d8","Type":"ContainerStarted","Data":"0c7c7833b08ab40337338ed5d5d245ad3e89a87473285029d562e19a7b765985"}
Jul 29 19:05:20.090361 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:20.090336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf47c5b58-l6zdl" event={"ID":"db5da208-6fa5-4048-9f63-974e985d6074","Type":"ContainerStarted","Data":"08e96c21ee192765b0a9f022259527182173e4a139c493e887a56160d16be6b2"}
Jul 29 19:05:20.107142 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:20.107091    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cf47c5b58-l6zdl" podStartSLOduration=1.178448625 podStartE2EDuration="5.107076248s" totalImagesPullingTime="3.928627623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:20.106864117 +0000 UTC m=+286.406536618" watchObservedRunningTime="2026-07-29 19:05:20.107076248 +0000 UTC m=+286.406748726"
Jul 29 19:05:22.100525 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:22.100482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2" event={"ID":"4a07e4ea-2748-4ca4-b93c-ba5a5652f012","Type":"ContainerStarted","Data":"fb7f7ddb53b8c02ea4fd3f2421d3e35226528f4d63e48a2e882f37882115010a"}
Jul 29 19:05:22.101018 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:22.100554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2" event={"ID":"4a07e4ea-2748-4ca4-b93c-ba5a5652f012","Type":"ContainerStarted","Data":"4d8cee8018cd80b2192a1abb17c6f3770100ad045d1723262735bb386bb4f8e2"}
Jul 29 19:05:22.119945 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:22.118948    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-mlnw2" podStartSLOduration=1.741827036 podStartE2EDuration="3.118925416s" totalImagesPullingTime="1.37709838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:22.118358693 +0000 UTC m=+288.418031192" watchObservedRunningTime="2026-07-29 19:05:22.118925416 +0000 UTC m=+288.418597920"
Jul 29 19:05:23.106435 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.106341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-79x22" event={"ID":"15228df5-30d8-4cd8-8de4-a55d961a28d8","Type":"ContainerStarted","Data":"d0c94bfd6d0620488d86fcb37fac52bc7472c7ae86346853488cd9cb589d1a07"}
Jul 29 19:05:23.106435 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.106398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-79x22" event={"ID":"15228df5-30d8-4cd8-8de4-a55d961a28d8","Type":"ContainerStarted","Data":"e04359f978cc37a66a17f5c0426f6a694f20ec7982b35d49f69bb5f79b489f25"}
Jul 29 19:05:23.123572 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.123516    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-79x22" podStartSLOduration=251.22129807 podStartE2EDuration="4m14.123495578s" totalImagesPullingTime="2.902197508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:23.122728069 +0000 UTC m=+289.422400569" watchObservedRunningTime="2026-07-29 19:05:23.123495578 +0000 UTC m=+289.423168084"
Jul 29 19:05:23.452550 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.452505    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-556988bd5-k5dv6"]
Jul 29 19:05:23.456182 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.456157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.464716 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.464671    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 19:05:23.467383 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.467342    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556988bd5-k5dv6"]
Jul 29 19:05:23.572984 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.572943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-trusted-ca-bundle\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.573222 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.573008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-config\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.573222 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.573038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2v2g\" (UniqueName: \"kubernetes.io/projected/59092cd6-88f8-4d12-add1-3f4371db1fb1-kube-api-access-b2v2g\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.573222 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.573067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-service-ca\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.573222 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.573116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-serving-cert\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.573222 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.573155    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-oauth-serving-cert\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.573490 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.573228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-oauth-config\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.649129 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.648420    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"]
Jul 29 19:05:23.652475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.652446    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:23.658170 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.657524    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 29 19:05:23.658170 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.657785    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 29 19:05:23.658170 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.658000    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-q9jxf\""
Jul 29 19:05:23.663840 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.661693    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"]
Jul 29 19:05:23.671618 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.668225    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8m5pg"]
Jul 29 19:05:23.674049 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.674017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-serving-cert\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.674230 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.674079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-oauth-serving-cert\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.674230 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.674132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-oauth-config\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.674230 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.674183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-trusted-ca-bundle\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.674406 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.674257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-config\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.674406 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.674281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2v2g\" (UniqueName: \"kubernetes.io/projected/59092cd6-88f8-4d12-add1-3f4371db1fb1-kube-api-access-b2v2g\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.674406 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.674309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-service-ca\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.675142 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.675117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-service-ca\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.675617 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.675593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.677811 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.675944    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-oauth-serving-cert\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.677811 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.676683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-config\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.677811 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.677174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-trusted-ca-bundle\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.681304 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.681256    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-serving-cert\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.682311 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.681818    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 19:05:23.682311 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.682091    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 19:05:23.682311 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.682127    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-58xhx\""
Jul 29 19:05:23.682551 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.682316    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 19:05:23.682982 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.682960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-oauth-config\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.685059 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.685039    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2v2g\" (UniqueName: \"kubernetes.io/projected/59092cd6-88f8-4d12-add1-3f4371db1fb1-kube-api-access-b2v2g\") pod \"console-556988bd5-k5dv6\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") " pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.768124 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.768027    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:23.775179 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.775145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-7fwhb\" (UID: \"9f163347-4a09-4a40-b14f-ebcf4e5f44c5\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:23.775345 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.775212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45473514-0e9c-4462-9784-8361953c92e4-sys\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.775345 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.775260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45473514-0e9c-4462-9784-8361953c92e4-root\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.775345 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.775313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45473514-0e9c-4462-9784-8361953c92e4-metrics-client-ca\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.775500 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.775390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-tls\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.775500 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.775429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.775578 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.775502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-7fwhb\" (UID: \"9f163347-4a09-4a40-b14f-ebcf4e5f44c5\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:23.775578 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.775549    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5vsh\" (UniqueName: \"kubernetes.io/projected/45473514-0e9c-4462-9784-8361953c92e4-kube-api-access-d5vsh\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.775663 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.775616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-7fwhb\" (UID: \"9f163347-4a09-4a40-b14f-ebcf4e5f44c5\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:23.775663 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.775647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-textfile\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.775808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.775671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.775808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.775698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jfd9\" (UniqueName: \"kubernetes.io/projected/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-kube-api-access-4jfd9\") pod \"openshift-state-metrics-b8f49f5c6-7fwhb\" (UID: \"9f163347-4a09-4a40-b14f-ebcf4e5f44c5\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:23.775808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.775744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-wtmp\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.876868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vsh\" (UniqueName: \"kubernetes.io/projected/45473514-0e9c-4462-9784-8361953c92e4-kube-api-access-d5vsh\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.876928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-7fwhb\" (UID: \"9f163347-4a09-4a40-b14f-ebcf4e5f44c5\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.876957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-textfile\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.876987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.877015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jfd9\" (UniqueName: \"kubernetes.io/projected/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-kube-api-access-4jfd9\") pod \"openshift-state-metrics-b8f49f5c6-7fwhb\" (UID: \"9f163347-4a09-4a40-b14f-ebcf4e5f44c5\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.877054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-wtmp\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.877098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-7fwhb\" (UID: \"9f163347-4a09-4a40-b14f-ebcf4e5f44c5\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.877132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45473514-0e9c-4462-9784-8361953c92e4-sys\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.877161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45473514-0e9c-4462-9784-8361953c92e4-root\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.877234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45473514-0e9c-4462-9784-8361953c92e4-metrics-client-ca\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.877270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-tls\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.877303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.877333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-7fwhb\" (UID: \"9f163347-4a09-4a40-b14f-ebcf4e5f44c5\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.877641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-wtmp\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.878370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45473514-0e9c-4462-9784-8361953c92e4-root\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:05:23.878657    2579 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 29 19:05:23.879850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.878855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45473514-0e9c-4462-9784-8361953c92e4-metrics-client-ca\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.880741 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:05:23.879159    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-openshift-state-metrics-tls podName:9f163347-4a09-4a40-b14f-ebcf4e5f44c5 nodeName:}" failed. No retries permitted until 2026-07-29 19:05:24.379137722 +0000 UTC m=+290.678810223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-7fwhb" (UID: "9f163347-4a09-4a40-b14f-ebcf4e5f44c5") : secret "openshift-state-metrics-tls" not found
Jul 29 19:05:23.880741 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:05:23.879272    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 19:05:23.880741 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:05:23.879320    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-tls podName:45473514-0e9c-4462-9784-8361953c92e4 nodeName:}" failed. No retries permitted until 2026-07-29 19:05:24.379306674 +0000 UTC m=+290.678979157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-tls") pod "node-exporter-8m5pg" (UID: "45473514-0e9c-4462-9784-8361953c92e4") : secret "node-exporter-tls" not found
Jul 29 19:05:23.880741 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.879365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45473514-0e9c-4462-9784-8361953c92e4-sys\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.880741 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.879738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.880741 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.880119    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-textfile\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.881452 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.881309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-7fwhb\" (UID: \"9f163347-4a09-4a40-b14f-ebcf4e5f44c5\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:23.883685 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.883656    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-7fwhb\" (UID: \"9f163347-4a09-4a40-b14f-ebcf4e5f44c5\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:23.885433 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.884462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.893010 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.892976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jfd9\" (UniqueName: \"kubernetes.io/projected/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-kube-api-access-4jfd9\") pod \"openshift-state-metrics-b8f49f5c6-7fwhb\" (UID: \"9f163347-4a09-4a40-b14f-ebcf4e5f44c5\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:23.896210 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.896135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vsh\" (UniqueName: \"kubernetes.io/projected/45473514-0e9c-4462-9784-8361953c92e4-kube-api-access-d5vsh\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:23.939260 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:23.939228    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556988bd5-k5dv6"]
Jul 29 19:05:23.942472 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:05:23.942405    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59092cd6_88f8_4d12_add1_3f4371db1fb1.slice/crio-558ed9ecbb6dbede7b9a337d43f0dd4d9fb96f45fbfc5f600efe351dcd2cc985 WatchSource:0}: Error finding container 558ed9ecbb6dbede7b9a337d43f0dd4d9fb96f45fbfc5f600efe351dcd2cc985: Status 404 returned error can't find the container with id 558ed9ecbb6dbede7b9a337d43f0dd4d9fb96f45fbfc5f600efe351dcd2cc985
Jul 29 19:05:24.112241 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:24.112175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556988bd5-k5dv6" event={"ID":"59092cd6-88f8-4d12-add1-3f4371db1fb1","Type":"ContainerStarted","Data":"8ec70a3e7c5e8eac803c4857b04875c7ebfd83bd35d434194067dcf84cd4da54"}
Jul 29 19:05:24.112850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:24.112257    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556988bd5-k5dv6" event={"ID":"59092cd6-88f8-4d12-add1-3f4371db1fb1","Type":"ContainerStarted","Data":"558ed9ecbb6dbede7b9a337d43f0dd4d9fb96f45fbfc5f600efe351dcd2cc985"}
Jul 29 19:05:24.112850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:24.112445    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-79x22"
Jul 29 19:05:24.134177 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:24.134117    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-556988bd5-k5dv6" podStartSLOduration=1.134096482 podStartE2EDuration="1.134096482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:24.132449771 +0000 UTC m=+290.432122270" watchObservedRunningTime="2026-07-29 19:05:24.134096482 +0000 UTC m=+290.433768981"
Jul 29 19:05:24.383076 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:24.383035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-7fwhb\" (UID: \"9f163347-4a09-4a40-b14f-ebcf4e5f44c5\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:24.383337 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:24.383225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-tls\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:24.386203 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:24.386162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45473514-0e9c-4462-9784-8361953c92e4-node-exporter-tls\") pod \"node-exporter-8m5pg\" (UID: \"45473514-0e9c-4462-9784-8361953c92e4\") " pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:24.386349 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:24.386256    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f163347-4a09-4a40-b14f-ebcf4e5f44c5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-7fwhb\" (UID: \"9f163347-4a09-4a40-b14f-ebcf4e5f44c5\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:24.597961 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:24.597915    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"
Jul 29 19:05:24.606485 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:24.606453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8m5pg"
Jul 29 19:05:24.625130 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:05:24.625095    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45473514_0e9c_4462_9784_8361953c92e4.slice/crio-3335b3810819611264944cb9587feab520791974977740e065a5110567cea20e WatchSource:0}: Error finding container 3335b3810819611264944cb9587feab520791974977740e065a5110567cea20e: Status 404 returned error can't find the container with id 3335b3810819611264944cb9587feab520791974977740e065a5110567cea20e
Jul 29 19:05:24.780960 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:24.780906    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb"]
Jul 29 19:05:24.784643 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:05:24.784605    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f163347_4a09_4a40_b14f_ebcf4e5f44c5.slice/crio-bacce85366f8a226234f4070f840f53be337d624999f97157b7e306bb38d7f99 WatchSource:0}: Error finding container bacce85366f8a226234f4070f840f53be337d624999f97157b7e306bb38d7f99: Status 404 returned error can't find the container with id bacce85366f8a226234f4070f840f53be337d624999f97157b7e306bb38d7f99
Jul 29 19:05:25.117780 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:25.117724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8m5pg" event={"ID":"45473514-0e9c-4462-9784-8361953c92e4","Type":"ContainerStarted","Data":"3335b3810819611264944cb9587feab520791974977740e065a5110567cea20e"}
Jul 29 19:05:25.120295 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:25.120232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb" event={"ID":"9f163347-4a09-4a40-b14f-ebcf4e5f44c5","Type":"ContainerStarted","Data":"67d4f6b63c7d1883504a5fc07ab661390d2c4bafd21d111df59b76d73fe89353"}
Jul 29 19:05:25.120445 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:25.120306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb" event={"ID":"9f163347-4a09-4a40-b14f-ebcf4e5f44c5","Type":"ContainerStarted","Data":"87125cb0466fc88d30d0efec425c67c0e9d1c1636101d3e31c475067e527b995"}
Jul 29 19:05:25.120445 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:25.120339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb" event={"ID":"9f163347-4a09-4a40-b14f-ebcf4e5f44c5","Type":"ContainerStarted","Data":"bacce85366f8a226234f4070f840f53be337d624999f97157b7e306bb38d7f99"}
Jul 29 19:05:25.607497 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:25.607450    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:25.607745 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:25.607543    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:25.614114 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:25.614044    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:26.134404 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:26.134373    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:05:28.191975 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:05:28.191926    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zm2md" podUID="1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42"
Jul 29 19:05:28.427994 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:28.427953    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g"]
Jul 29 19:05:28.431740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:28.431713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g"
Jul 29 19:05:28.434470 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:28.434425    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 29 19:05:28.434664 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:28.434426    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fq9d7\""
Jul 29 19:05:28.439141 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:28.438578    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g"]
Jul 29 19:05:28.527168 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:28.527067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab7d3a5-50ac-4760-9486-825bbeaee987-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pn52g\" (UID: \"6ab7d3a5-50ac-4760-9486-825bbeaee987\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g"
Jul 29 19:05:28.527369 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:28.527233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82t8v\" (UniqueName: \"kubernetes.io/projected/6ab7d3a5-50ac-4760-9486-825bbeaee987-kube-api-access-82t8v\") pod \"monitoring-plugin-68b7fc9574-pn52g\" (UID: \"6ab7d3a5-50ac-4760-9486-825bbeaee987\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g"
Jul 29 19:05:28.627755 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:28.627710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82t8v\" (UniqueName: \"kubernetes.io/projected/6ab7d3a5-50ac-4760-9486-825bbeaee987-kube-api-access-82t8v\") pod \"monitoring-plugin-68b7fc9574-pn52g\" (UID: \"6ab7d3a5-50ac-4760-9486-825bbeaee987\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g"
Jul 29 19:05:28.627972 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:28.627790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab7d3a5-50ac-4760-9486-825bbeaee987-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pn52g\" (UID: \"6ab7d3a5-50ac-4760-9486-825bbeaee987\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g"
Jul 29 19:05:28.630812 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:28.630769    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ab7d3a5-50ac-4760-9486-825bbeaee987-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pn52g\" (UID: \"6ab7d3a5-50ac-4760-9486-825bbeaee987\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g"
Jul 29 19:05:28.636071 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:28.636041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82t8v\" (UniqueName: \"kubernetes.io/projected/6ab7d3a5-50ac-4760-9486-825bbeaee987-kube-api-access-82t8v\") pod \"monitoring-plugin-68b7fc9574-pn52g\" (UID: \"6ab7d3a5-50ac-4760-9486-825bbeaee987\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g"
Jul 29 19:05:28.744748 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:28.744690    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g"
Jul 29 19:05:29.447981 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.447946    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556988bd5-k5dv6"]
Jul 29 19:05:29.878881 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.878770    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:05:29.969360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.969324    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:05:29.969547 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.969486    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:29.972439 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.972405    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 19:05:29.972693 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.972415    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 19:05:29.972693 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.972552    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 19:05:29.972874 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.972789    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 19:05:29.972874 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.972798    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 19:05:29.972874 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.972818    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 19:05:29.973057 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.972890    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 19:05:29.973840 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.973683    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 19:05:29.975885 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.975863    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fvmbc147o7smk\""
Jul 29 19:05:29.978242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.978094    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 19:05:29.979719 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.979148    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-whlfj\""
Jul 29 19:05:29.979719 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.979474    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 19:05:29.980431 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.980154    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 19:05:29.981079 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.980952    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 19:05:29.981815 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.981347    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 19:05:29.984339 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.984317    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 19:05:29.992777 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:29.992744    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 19:05:30.144541 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.144394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-config\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.144541 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.144445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-web-config\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.144541 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.144481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.144541 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.144509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2ltk\" (UniqueName: \"kubernetes.io/projected/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-kube-api-access-m2ltk\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.144598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.144670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.144705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.144735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.144772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.144799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.144879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.144905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.144943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.144981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.145007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145532 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.145055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145532 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.145096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145532 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.145122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145532 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.145237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.145532 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.145293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-config-out\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246080 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246080 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246408 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246408 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246408 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246408 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246408 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246408 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246408 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246408 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-config-out\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246408 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-config\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246846 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-web-config\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246846 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246846 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2ltk\" (UniqueName: \"kubernetes.io/projected/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-kube-api-access-m2ltk\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246846 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246846 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246846 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246846 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246846 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.246846 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.246668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.247280 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.247235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.247854 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.247502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.247854 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.247694    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.247854 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.247711    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.248574 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.248543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.250386 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.249713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.250386 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.250104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.250386 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.250334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.250823 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.250611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.251781 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.251353    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-config-out\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.251781 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.251369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.251781 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.251729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.252360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.252243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.252677 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.252598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.252677 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.252669    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.253993 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.253971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.254298 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.254279    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.254841 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.254787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-web-config\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.254841 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.254810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-config\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.257563 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.257545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2ltk\" (UniqueName: \"kubernetes.io/projected/2f1b718a-1bde-4539-aaa0-8ce652dc5d18-kube-api-access-m2ltk\") pod \"prometheus-k8s-0\" (UID: \"2f1b718a-1bde-4539-aaa0-8ce652dc5d18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:30.289678 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:30.289636    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:33.297745 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:33.297707    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:05:33.312350 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:33.311350    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g"]
Jul 29 19:05:33.508072 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:05:33.508029    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab7d3a5_50ac_4760_9486_825bbeaee987.slice/crio-b6d9231294f65ce4b7d79a025ab6d946e4a9b712c0211defd0783fe06258f096 WatchSource:0}: Error finding container b6d9231294f65ce4b7d79a025ab6d946e4a9b712c0211defd0783fe06258f096: Status 404 returned error can't find the container with id b6d9231294f65ce4b7d79a025ab6d946e4a9b712c0211defd0783fe06258f096
Jul 29 19:05:33.508888 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:05:33.508764    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1b718a_1bde_4539_aaa0_8ce652dc5d18.slice/crio-b29f60f21ab1718017ad879f779d539a1c018226c336e65355cffd261bf24779 WatchSource:0}: Error finding container b29f60f21ab1718017ad879f779d539a1c018226c336e65355cffd261bf24779: Status 404 returned error can't find the container with id b29f60f21ab1718017ad879f779d539a1c018226c336e65355cffd261bf24779
Jul 29 19:05:33.768964 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:33.768936    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:34.072233 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:34.072175    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:05:34.073243 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:34.073215    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:05:34.102288 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:34.102248    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 19:05:34.123782 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:34.123727    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-79x22"
Jul 29 19:05:34.173084 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:34.173045    2579 generic.go:356] "Generic (PLEG): container finished" podID="45473514-0e9c-4462-9784-8361953c92e4" containerID="c6ff823c7df7aca3bcee47b07ee4e5d3549feef8dbfb0c1873a28804cfdafbf6" exitCode=0
Jul 29 19:05:34.173367 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:34.173136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8m5pg" event={"ID":"45473514-0e9c-4462-9784-8361953c92e4","Type":"ContainerDied","Data":"c6ff823c7df7aca3bcee47b07ee4e5d3549feef8dbfb0c1873a28804cfdafbf6"}
Jul 29 19:05:34.183988 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:34.183940    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f1b718a-1bde-4539-aaa0-8ce652dc5d18","Type":"ContainerStarted","Data":"b29f60f21ab1718017ad879f779d539a1c018226c336e65355cffd261bf24779"}
Jul 29 19:05:34.209563 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:34.209013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g" event={"ID":"6ab7d3a5-50ac-4760-9486-825bbeaee987","Type":"ContainerStarted","Data":"b6d9231294f65ce4b7d79a025ab6d946e4a9b712c0211defd0783fe06258f096"}
Jul 29 19:05:34.212462 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:34.212418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-lvx6t" event={"ID":"47785f11-d0a3-4108-a84c-71da470dca6c","Type":"ContainerStarted","Data":"113b26053fc23419c0c9cf80e0b12a5e4864a8663e457f596cd8c33dc0c87303"}
Jul 29 19:05:34.220305 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:34.220167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb" event={"ID":"9f163347-4a09-4a40-b14f-ebcf4e5f44c5","Type":"ContainerStarted","Data":"16b6b61668787b317c9da85acf6b85b34e0c76f056ec885eff9656e640f282c3"}
Jul 29 19:05:34.236524 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:34.236237    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-lvx6t"
Jul 29 19:05:34.250182 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:34.250150    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-lvx6t"
Jul 29 19:05:34.256493 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:34.254702    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-lvx6t" podStartSLOduration=2.03685423 podStartE2EDuration="24.254680728s" totalImagesPullingTime="22.217826498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:34.252503737 +0000 UTC m=+300.552176248" watchObservedRunningTime="2026-07-29 19:05:34.254680728 +0000 UTC m=+300.554353228"
Jul 29 19:05:34.270824 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:34.270756    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7fwhb" podStartSLOduration=2.473699994 podStartE2EDuration="11.270732518s" totalImagesPullingTime="8.797032524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:34.270424199 +0000 UTC m=+300.570096701" watchObservedRunningTime="2026-07-29 19:05:34.270732518 +0000 UTC m=+300.570405018"
Jul 29 19:05:35.230618 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:35.230566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8m5pg" event={"ID":"45473514-0e9c-4462-9784-8361953c92e4","Type":"ContainerStarted","Data":"a098f512dfd12d72cd31b41ed4ac56568f9b5cd4e50d7e30cd747e0e82f61c1e"}
Jul 29 19:05:35.231131 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:35.230644    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8m5pg" event={"ID":"45473514-0e9c-4462-9784-8361953c92e4","Type":"ContainerStarted","Data":"568feee451a9a664066e5f503588c496207a519c3008381234b65bb08ffb4918"}
Jul 29 19:05:35.250967 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:35.250893    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8m5pg" podStartSLOduration=3.664701248 podStartE2EDuration="12.250866304s" totalImagesPullingTime="8.486155759s" totalInitContainerRuntime="100.009297ms" isStatefulPod=true podCreationTimestamp="2026-07-29 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:35.249414498 +0000 UTC m=+301.549086997" watchObservedRunningTime="2026-07-29 19:05:35.250866304 +0000 UTC m=+301.550538805"
Jul 29 19:05:36.241256 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:36.238685    2579 generic.go:356] "Generic (PLEG): container finished" podID="2f1b718a-1bde-4539-aaa0-8ce652dc5d18" containerID="df0d1596d7e3ca7d24db36c06ca579c194e27b2b3aa8c9d7d1e3225c755c342c" exitCode=0
Jul 29 19:05:36.241256 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:36.240280    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f1b718a-1bde-4539-aaa0-8ce652dc5d18","Type":"ContainerDied","Data":"df0d1596d7e3ca7d24db36c06ca579c194e27b2b3aa8c9d7d1e3225c755c342c"}
Jul 29 19:05:36.242294 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:36.242272    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:05:37.245493 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:37.245393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g" event={"ID":"6ab7d3a5-50ac-4760-9486-825bbeaee987","Type":"ContainerStarted","Data":"27e57512cfb6812a4625a0dd953e4812b2a8dcf5bdcc69a4ee305cd471688991"}
Jul 29 19:05:37.245992 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:37.245598    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g"
Jul 29 19:05:37.252142 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:37.252112    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g"
Jul 29 19:05:37.262874 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:37.262817    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pn52g" podStartSLOduration=5.890441924 podStartE2EDuration="9.262767722s" totalImagesPullingTime="3.372325798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:37.262430417 +0000 UTC m=+303.562102909" watchObservedRunningTime="2026-07-29 19:05:37.262767722 +0000 UTC m=+303.562440221"
Jul 29 19:05:40.192415 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:40.192369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:05:40.672489 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:40.671657    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cf47c5b58-l6zdl"]
Jul 29 19:05:42.278703 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:42.278527    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f1b718a-1bde-4539-aaa0-8ce652dc5d18","Type":"ContainerStarted","Data":"65e44e6bd1ced3c02fd024e621ee743a8bab8772e26b987872e1dd0fc9eb6ccb"}
Jul 29 19:05:42.278703 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:42.278627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f1b718a-1bde-4539-aaa0-8ce652dc5d18","Type":"ContainerStarted","Data":"4443444fd62f3eade0fe54c0d32c682633c4945df503779de6094f0f7af4a70b"}
Jul 29 19:05:45.297241 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:45.296512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f1b718a-1bde-4539-aaa0-8ce652dc5d18","Type":"ContainerStarted","Data":"935c622a2eaf568f93d5c66da54f10cce150e2482e955a08d139b7ea1e002b91"}
Jul 29 19:05:45.297241 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:45.296663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f1b718a-1bde-4539-aaa0-8ce652dc5d18","Type":"ContainerStarted","Data":"df9df5925d9ed24709d57ccd6152af63ee2868ffb0948880cea5ddc40a7e1159"}
Jul 29 19:05:45.297241 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:45.296708    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f1b718a-1bde-4539-aaa0-8ce652dc5d18","Type":"ContainerStarted","Data":"404afe8166d0adcd5272a68981932b1e0a1d3a177699a48dc7b44ada438b39cc"}
Jul 29 19:05:45.297241 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:45.296748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f1b718a-1bde-4539-aaa0-8ce652dc5d18","Type":"ContainerStarted","Data":"0a287b59c2ce5b2b4b32bd96db9459459a00398830ce1298e1313069e6b495ea"}
Jul 29 19:05:45.325867 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:45.325800    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.453501984 podStartE2EDuration="16.325776442s" totalImagesPullingTime="9.834990024s" totalInitContainerRuntime="37.284434ms" isStatefulPod=false podCreationTimestamp="2026-07-29 19:05:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:45.323573868 +0000 UTC m=+311.623246422" watchObservedRunningTime="2026-07-29 19:05:45.325776442 +0000 UTC m=+311.625448943"
Jul 29 19:05:50.290277 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:50.290229    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:05:54.471108 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.471034    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-556988bd5-k5dv6" podUID="59092cd6-88f8-4d12-add1-3f4371db1fb1" containerName="console" containerID="cri-o://8ec70a3e7c5e8eac803c4857b04875c7ebfd83bd35d434194067dcf84cd4da54" gracePeriod=15
Jul 29 19:05:54.757258 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.757223    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:54.925128 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.925089    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-oauth-config\") pod \"59092cd6-88f8-4d12-add1-3f4371db1fb1\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") "
Jul 29 19:05:54.925128 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.925130    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-service-ca\") pod \"59092cd6-88f8-4d12-add1-3f4371db1fb1\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") "
Jul 29 19:05:54.925491 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.925157    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-serving-cert\") pod \"59092cd6-88f8-4d12-add1-3f4371db1fb1\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") "
Jul 29 19:05:54.925491 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.925211    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-trusted-ca-bundle\") pod \"59092cd6-88f8-4d12-add1-3f4371db1fb1\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") "
Jul 29 19:05:54.925491 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.925256    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2v2g\" (UniqueName: \"kubernetes.io/projected/59092cd6-88f8-4d12-add1-3f4371db1fb1-kube-api-access-b2v2g\") pod \"59092cd6-88f8-4d12-add1-3f4371db1fb1\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") "
Jul 29 19:05:54.925491 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.925281    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-oauth-serving-cert\") pod \"59092cd6-88f8-4d12-add1-3f4371db1fb1\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") "
Jul 29 19:05:54.925491 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.925342    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-config\") pod \"59092cd6-88f8-4d12-add1-3f4371db1fb1\" (UID: \"59092cd6-88f8-4d12-add1-3f4371db1fb1\") "
Jul 29 19:05:54.925777 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.925733    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "59092cd6-88f8-4d12-add1-3f4371db1fb1" (UID: "59092cd6-88f8-4d12-add1-3f4371db1fb1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:05:54.925971 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.925939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-config" (OuterVolumeSpecName: "console-config") pod "59092cd6-88f8-4d12-add1-3f4371db1fb1" (UID: "59092cd6-88f8-4d12-add1-3f4371db1fb1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:05:54.926106 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.925956    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "59092cd6-88f8-4d12-add1-3f4371db1fb1" (UID: "59092cd6-88f8-4d12-add1-3f4371db1fb1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:05:54.926210 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.926162    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-service-ca" (OuterVolumeSpecName: "service-ca") pod "59092cd6-88f8-4d12-add1-3f4371db1fb1" (UID: "59092cd6-88f8-4d12-add1-3f4371db1fb1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:05:54.927925 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.927897    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "59092cd6-88f8-4d12-add1-3f4371db1fb1" (UID: "59092cd6-88f8-4d12-add1-3f4371db1fb1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:05:54.928012 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.927939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59092cd6-88f8-4d12-add1-3f4371db1fb1-kube-api-access-b2v2g" (OuterVolumeSpecName: "kube-api-access-b2v2g") pod "59092cd6-88f8-4d12-add1-3f4371db1fb1" (UID: "59092cd6-88f8-4d12-add1-3f4371db1fb1"). InnerVolumeSpecName "kube-api-access-b2v2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:05:54.928012 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:54.927937    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "59092cd6-88f8-4d12-add1-3f4371db1fb1" (UID: "59092cd6-88f8-4d12-add1-3f4371db1fb1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:05:55.027126 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.027013    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2v2g\" (UniqueName: \"kubernetes.io/projected/59092cd6-88f8-4d12-add1-3f4371db1fb1-kube-api-access-b2v2g\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:55.027126 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.027054    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-oauth-serving-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:55.027126 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.027063    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-config\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:55.027126 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.027074    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-oauth-config\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:55.027126 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.027083    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:55.027126 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.027091    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59092cd6-88f8-4d12-add1-3f4371db1fb1-console-serving-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:55.027126 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.027099    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59092cd6-88f8-4d12-add1-3f4371db1fb1-trusted-ca-bundle\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:05:55.340185 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.340098    2579 generic.go:356] "Generic (PLEG): container finished" podID="59092cd6-88f8-4d12-add1-3f4371db1fb1" containerID="8ec70a3e7c5e8eac803c4857b04875c7ebfd83bd35d434194067dcf84cd4da54" exitCode=0
Jul 29 19:05:55.340185 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.340153    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556988bd5-k5dv6"
Jul 29 19:05:55.340475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.340186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556988bd5-k5dv6" event={"ID":"59092cd6-88f8-4d12-add1-3f4371db1fb1","Type":"ContainerDied","Data":"8ec70a3e7c5e8eac803c4857b04875c7ebfd83bd35d434194067dcf84cd4da54"}
Jul 29 19:05:55.340475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.340273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556988bd5-k5dv6" event={"ID":"59092cd6-88f8-4d12-add1-3f4371db1fb1","Type":"ContainerDied","Data":"558ed9ecbb6dbede7b9a337d43f0dd4d9fb96f45fbfc5f600efe351dcd2cc985"}
Jul 29 19:05:55.340475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.340301    2579 scope.go:117] "RemoveContainer" containerID="8ec70a3e7c5e8eac803c4857b04875c7ebfd83bd35d434194067dcf84cd4da54"
Jul 29 19:05:55.341935 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.341911    2579 generic.go:356] "Generic (PLEG): container finished" podID="1a486fe1-02df-4c80-8b23-83cb3e070a5a" containerID="88d78abd6c17d03c3f1542a66be0aed67286defb46cebd6ec6e22ab2efcc7866" exitCode=0
Jul 29 19:05:55.342052 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.341951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569" event={"ID":"1a486fe1-02df-4c80-8b23-83cb3e070a5a","Type":"ContainerDied","Data":"88d78abd6c17d03c3f1542a66be0aed67286defb46cebd6ec6e22ab2efcc7866"}
Jul 29 19:05:55.342284 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.342270    2579 scope.go:117] "RemoveContainer" containerID="88d78abd6c17d03c3f1542a66be0aed67286defb46cebd6ec6e22ab2efcc7866"
Jul 29 19:05:55.350491 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.350475    2579 scope.go:117] "RemoveContainer" containerID="8ec70a3e7c5e8eac803c4857b04875c7ebfd83bd35d434194067dcf84cd4da54"
Jul 29 19:05:55.350785 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:05:55.350755    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec70a3e7c5e8eac803c4857b04875c7ebfd83bd35d434194067dcf84cd4da54\": container with ID starting with 8ec70a3e7c5e8eac803c4857b04875c7ebfd83bd35d434194067dcf84cd4da54 not found: ID does not exist" containerID="8ec70a3e7c5e8eac803c4857b04875c7ebfd83bd35d434194067dcf84cd4da54"
Jul 29 19:05:55.350853 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.350789    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec70a3e7c5e8eac803c4857b04875c7ebfd83bd35d434194067dcf84cd4da54"} err="failed to get container status \"8ec70a3e7c5e8eac803c4857b04875c7ebfd83bd35d434194067dcf84cd4da54\": rpc error: code = NotFound desc = could not find container \"8ec70a3e7c5e8eac803c4857b04875c7ebfd83bd35d434194067dcf84cd4da54\": container with ID starting with 8ec70a3e7c5e8eac803c4857b04875c7ebfd83bd35d434194067dcf84cd4da54 not found: ID does not exist"
Jul 29 19:05:55.373607 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.373579    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556988bd5-k5dv6"]
Jul 29 19:05:55.379278 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:55.379244    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-556988bd5-k5dv6"]
Jul 29 19:05:56.195403 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:56.195359    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59092cd6-88f8-4d12-add1-3f4371db1fb1" path="/var/lib/kubelet/pods/59092cd6-88f8-4d12-add1-3f4371db1fb1/volumes"
Jul 29 19:05:56.353712 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:05:56.353666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ds569" event={"ID":"1a486fe1-02df-4c80-8b23-83cb3e070a5a","Type":"ContainerStarted","Data":"51f46e05a0d13bce4adf43a51d1fffabb50cfd692d164c05d7b80048fb1c3328"}
Jul 29 19:06:00.371261 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:00.371223    2579 generic.go:356] "Generic (PLEG): container finished" podID="30cfb52e-9435-492b-b39d-f6ab737038bd" containerID="f96b61398f76d7313b992ac3d135483ec83b9d309a7638d77c80cea83a39ca3d" exitCode=0
Jul 29 19:06:00.371792 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:00.371222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld" event={"ID":"30cfb52e-9435-492b-b39d-f6ab737038bd","Type":"ContainerDied","Data":"f96b61398f76d7313b992ac3d135483ec83b9d309a7638d77c80cea83a39ca3d"}
Jul 29 19:06:00.371792 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:00.371673    2579 scope.go:117] "RemoveContainer" containerID="f96b61398f76d7313b992ac3d135483ec83b9d309a7638d77c80cea83a39ca3d"
Jul 29 19:06:00.372962 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:00.372941    2579 generic.go:356] "Generic (PLEG): container finished" podID="68123628-c31f-4b00-b091-0c1e8802972a" containerID="2b278102c93dc62701c08f558a4c8ee94649457a687075b98f473d871900cde6" exitCode=0
Jul 29 19:06:00.373068 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:00.373017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-45jh7" event={"ID":"68123628-c31f-4b00-b091-0c1e8802972a","Type":"ContainerDied","Data":"2b278102c93dc62701c08f558a4c8ee94649457a687075b98f473d871900cde6"}
Jul 29 19:06:00.373380 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:00.373364    2579 scope.go:117] "RemoveContainer" containerID="2b278102c93dc62701c08f558a4c8ee94649457a687075b98f473d871900cde6"
Jul 29 19:06:01.378729 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:01.378685    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-45jh7" event={"ID":"68123628-c31f-4b00-b091-0c1e8802972a","Type":"ContainerStarted","Data":"97b925fc475cb09bb434d7f88eea2115a62210a6a62a124775e85deb93ec1c2e"}
Jul 29 19:06:01.380580 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:01.380542    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld" event={"ID":"30cfb52e-9435-492b-b39d-f6ab737038bd","Type":"ContainerStarted","Data":"4df7f6d3669b70c5ea07ca1fcf7fdbd9bedd0464083e65a8be035fa83b44cd0a"}
Jul 29 19:06:05.700084 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:05.700017    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cf47c5b58-l6zdl" podUID="db5da208-6fa5-4048-9f63-974e985d6074" containerName="console" containerID="cri-o://08e96c21ee192765b0a9f022259527182173e4a139c493e887a56160d16be6b2" gracePeriod=15
Jul 29 19:06:05.972341 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:05.972267    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:06:06.025219 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.025159    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db5da208-6fa5-4048-9f63-974e985d6074-console-serving-cert\") pod \"db5da208-6fa5-4048-9f63-974e985d6074\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") "
Jul 29 19:06:06.025434 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.025232    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db5da208-6fa5-4048-9f63-974e985d6074-console-oauth-config\") pod \"db5da208-6fa5-4048-9f63-974e985d6074\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") "
Jul 29 19:06:06.025434 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.025262    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhklv\" (UniqueName: \"kubernetes.io/projected/db5da208-6fa5-4048-9f63-974e985d6074-kube-api-access-xhklv\") pod \"db5da208-6fa5-4048-9f63-974e985d6074\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") "
Jul 29 19:06:06.025434 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.025318    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-console-config\") pod \"db5da208-6fa5-4048-9f63-974e985d6074\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") "
Jul 29 19:06:06.025434 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.025349    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-service-ca\") pod \"db5da208-6fa5-4048-9f63-974e985d6074\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") "
Jul 29 19:06:06.025434 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.025383    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-oauth-serving-cert\") pod \"db5da208-6fa5-4048-9f63-974e985d6074\" (UID: \"db5da208-6fa5-4048-9f63-974e985d6074\") "
Jul 29 19:06:06.025831 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.025798    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-service-ca" (OuterVolumeSpecName: "service-ca") pod "db5da208-6fa5-4048-9f63-974e985d6074" (UID: "db5da208-6fa5-4048-9f63-974e985d6074"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:06:06.025831 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.025810    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "db5da208-6fa5-4048-9f63-974e985d6074" (UID: "db5da208-6fa5-4048-9f63-974e985d6074"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:06:06.025947 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.025804    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-console-config" (OuterVolumeSpecName: "console-config") pod "db5da208-6fa5-4048-9f63-974e985d6074" (UID: "db5da208-6fa5-4048-9f63-974e985d6074"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:06:06.027822 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.027800    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5da208-6fa5-4048-9f63-974e985d6074-kube-api-access-xhklv" (OuterVolumeSpecName: "kube-api-access-xhklv") pod "db5da208-6fa5-4048-9f63-974e985d6074" (UID: "db5da208-6fa5-4048-9f63-974e985d6074"). InnerVolumeSpecName "kube-api-access-xhklv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:06:06.028056 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.028040    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5da208-6fa5-4048-9f63-974e985d6074-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "db5da208-6fa5-4048-9f63-974e985d6074" (UID: "db5da208-6fa5-4048-9f63-974e985d6074"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:06:06.028127 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.028115    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5da208-6fa5-4048-9f63-974e985d6074-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "db5da208-6fa5-4048-9f63-974e985d6074" (UID: "db5da208-6fa5-4048-9f63-974e985d6074"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:06:06.127039 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.126997    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db5da208-6fa5-4048-9f63-974e985d6074-console-serving-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:06:06.127039 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.127029    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db5da208-6fa5-4048-9f63-974e985d6074-console-oauth-config\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:06:06.127039 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.127038    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhklv\" (UniqueName: \"kubernetes.io/projected/db5da208-6fa5-4048-9f63-974e985d6074-kube-api-access-xhklv\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:06:06.127039 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.127047    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-console-config\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:06:06.127366 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.127056    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:06:06.127366 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.127064    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db5da208-6fa5-4048-9f63-974e985d6074-oauth-serving-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:06:06.403707 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.403663    2579 generic.go:356] "Generic (PLEG): container finished" podID="db5da208-6fa5-4048-9f63-974e985d6074" containerID="08e96c21ee192765b0a9f022259527182173e4a139c493e887a56160d16be6b2" exitCode=0
Jul 29 19:06:06.403707 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.403708    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf47c5b58-l6zdl"
Jul 29 19:06:06.404028 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.403710    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf47c5b58-l6zdl" event={"ID":"db5da208-6fa5-4048-9f63-974e985d6074","Type":"ContainerDied","Data":"08e96c21ee192765b0a9f022259527182173e4a139c493e887a56160d16be6b2"}
Jul 29 19:06:06.404028 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.403783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf47c5b58-l6zdl" event={"ID":"db5da208-6fa5-4048-9f63-974e985d6074","Type":"ContainerDied","Data":"fc58b01e18c45328578ad63a2d96659e1ae4583a64fa2951183fe2efb644efca"}
Jul 29 19:06:06.404028 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.403813    2579 scope.go:117] "RemoveContainer" containerID="08e96c21ee192765b0a9f022259527182173e4a139c493e887a56160d16be6b2"
Jul 29 19:06:06.412927 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.412910    2579 scope.go:117] "RemoveContainer" containerID="08e96c21ee192765b0a9f022259527182173e4a139c493e887a56160d16be6b2"
Jul 29 19:06:06.413239 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:06:06.413221    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e96c21ee192765b0a9f022259527182173e4a139c493e887a56160d16be6b2\": container with ID starting with 08e96c21ee192765b0a9f022259527182173e4a139c493e887a56160d16be6b2 not found: ID does not exist" containerID="08e96c21ee192765b0a9f022259527182173e4a139c493e887a56160d16be6b2"
Jul 29 19:06:06.413300 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.413249    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e96c21ee192765b0a9f022259527182173e4a139c493e887a56160d16be6b2"} err="failed to get container status \"08e96c21ee192765b0a9f022259527182173e4a139c493e887a56160d16be6b2\": rpc error: code = NotFound desc = could not find container \"08e96c21ee192765b0a9f022259527182173e4a139c493e887a56160d16be6b2\": container with ID starting with 08e96c21ee192765b0a9f022259527182173e4a139c493e887a56160d16be6b2 not found: ID does not exist"
Jul 29 19:06:06.420946 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.420919    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cf47c5b58-l6zdl"]
Jul 29 19:06:06.422583 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:06.422563    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cf47c5b58-l6zdl"]
Jul 29 19:06:08.194787 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:08.194751    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db5da208-6fa5-4048-9f63-974e985d6074" path="/var/lib/kubelet/pods/db5da208-6fa5-4048-9f63-974e985d6074/volumes"
Jul 29 19:06:30.290393 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:30.290278    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:06:30.310980 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:30.310953    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:06:30.517416 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:30.517389    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:06:47.198084 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:47.198020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:06:47.200669 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:47.200642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42-metrics-certs\") pod \"network-metrics-daemon-zm2md\" (UID: \"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42\") " pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:06:47.395809 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:47.395779    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qh2tz\""
Jul 29 19:06:47.403256 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:47.403218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zm2md"
Jul 29 19:06:47.534732 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:47.534707    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zm2md"]
Jul 29 19:06:47.537718 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:06:47.537689    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6cd626_3e5e_4fcc_a9c8_0a7cd4513d42.slice/crio-53040a783660e082462224ef3c05a33dddadfcf4f468ad575ab4f94e9e03db89 WatchSource:0}: Error finding container 53040a783660e082462224ef3c05a33dddadfcf4f468ad575ab4f94e9e03db89: Status 404 returned error can't find the container with id 53040a783660e082462224ef3c05a33dddadfcf4f468ad575ab4f94e9e03db89
Jul 29 19:06:47.567475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:47.567437    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zm2md" event={"ID":"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42","Type":"ContainerStarted","Data":"53040a783660e082462224ef3c05a33dddadfcf4f468ad575ab4f94e9e03db89"}
Jul 29 19:06:49.577533 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:49.577438    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zm2md" event={"ID":"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42","Type":"ContainerStarted","Data":"9a8e4b7805f7503c2e0fc34214dc792e1c0039bd576d31758558db83b8ebc505"}
Jul 29 19:06:49.577533 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:49.577495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zm2md" event={"ID":"1c6cd626-3e5e-4fcc-a9c8-0a7cd4513d42","Type":"ContainerStarted","Data":"8df2cf2da0f14cd355b050e7d8b5cfd3e231995c9a9ee56a312ddeec54f1efa1"}
Jul 29 19:06:49.595235 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:06:49.595152    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zm2md" podStartSLOduration=373.878520447 podStartE2EDuration="6m15.595134231s" totalImagesPullingTime="1.716613784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:06:49.594741542 +0000 UTC m=+375.894414054" watchObservedRunningTime="2026-07-29 19:06:49.595134231 +0000 UTC m=+375.894806785"
Jul 29 19:08:48.133581 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.133542    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzc42"]
Jul 29 19:08:48.134043 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.133860    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db5da208-6fa5-4048-9f63-974e985d6074" containerName="console"
Jul 29 19:08:48.134043 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.133871    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="db5da208-6fa5-4048-9f63-974e985d6074" containerName="console"
Jul 29 19:08:48.134043 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.133882    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59092cd6-88f8-4d12-add1-3f4371db1fb1" containerName="console"
Jul 29 19:08:48.134043 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.133887    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="59092cd6-88f8-4d12-add1-3f4371db1fb1" containerName="console"
Jul 29 19:08:48.134043 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.133970    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="59092cd6-88f8-4d12-add1-3f4371db1fb1" containerName="console"
Jul 29 19:08:48.134043 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.133980    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="db5da208-6fa5-4048-9f63-974e985d6074" containerName="console"
Jul 29 19:08:48.135909 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.135891    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzc42"
Jul 29 19:08:48.138811 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.138780    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 19:08:48.138976 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.138827    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:08:48.140154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.140131    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-hcvbm\""
Jul 29 19:08:48.148974 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.148948    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzc42"]
Jul 29 19:08:48.285154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.285106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/503e0dee-5dd3-468b-ac63-0883d7ae43b8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kzc42\" (UID: \"503e0dee-5dd3-468b-ac63-0883d7ae43b8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzc42"
Jul 29 19:08:48.285154 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.285156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpbfz\" (UniqueName: \"kubernetes.io/projected/503e0dee-5dd3-468b-ac63-0883d7ae43b8-kube-api-access-qpbfz\") pod \"openshift-lws-operator-fd8ccff4c-kzc42\" (UID: \"503e0dee-5dd3-468b-ac63-0883d7ae43b8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzc42"
Jul 29 19:08:48.386541 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.386436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/503e0dee-5dd3-468b-ac63-0883d7ae43b8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kzc42\" (UID: \"503e0dee-5dd3-468b-ac63-0883d7ae43b8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzc42"
Jul 29 19:08:48.386541 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.386481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbfz\" (UniqueName: \"kubernetes.io/projected/503e0dee-5dd3-468b-ac63-0883d7ae43b8-kube-api-access-qpbfz\") pod \"openshift-lws-operator-fd8ccff4c-kzc42\" (UID: \"503e0dee-5dd3-468b-ac63-0883d7ae43b8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzc42"
Jul 29 19:08:48.386914 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.386892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/503e0dee-5dd3-468b-ac63-0883d7ae43b8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kzc42\" (UID: \"503e0dee-5dd3-468b-ac63-0883d7ae43b8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzc42"
Jul 29 19:08:48.395594 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.395568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpbfz\" (UniqueName: \"kubernetes.io/projected/503e0dee-5dd3-468b-ac63-0883d7ae43b8-kube-api-access-qpbfz\") pod \"openshift-lws-operator-fd8ccff4c-kzc42\" (UID: \"503e0dee-5dd3-468b-ac63-0883d7ae43b8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzc42"
Jul 29 19:08:48.457065 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.457022    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzc42"
Jul 29 19:08:48.602250 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:48.602185    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzc42"]
Jul 29 19:08:48.606404 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:08:48.606377    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod503e0dee_5dd3_468b_ac63_0883d7ae43b8.slice/crio-32bb5beb578ecee88413815990ec86127f91225f990c83349dc424c74cf6f347 WatchSource:0}: Error finding container 32bb5beb578ecee88413815990ec86127f91225f990c83349dc424c74cf6f347: Status 404 returned error can't find the container with id 32bb5beb578ecee88413815990ec86127f91225f990c83349dc424c74cf6f347
Jul 29 19:08:49.084688 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:49.084468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzc42" event={"ID":"503e0dee-5dd3-468b-ac63-0883d7ae43b8","Type":"ContainerStarted","Data":"32bb5beb578ecee88413815990ec86127f91225f990c83349dc424c74cf6f347"}
Jul 29 19:08:51.094763 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:51.094662    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzc42" event={"ID":"503e0dee-5dd3-468b-ac63-0883d7ae43b8","Type":"ContainerStarted","Data":"bf83efbaa28c83547aa480d1e7e7b9ccdc3bfaa49d1ac82bf2c13c0898ea0f55"}
Jul 29 19:08:51.123657 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:08:51.123592    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzc42" podStartSLOduration=0.905879236 podStartE2EDuration="3.123549234s" totalImagesPullingTime="2.217669998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:08:51.122016847 +0000 UTC m=+497.421689348" watchObservedRunningTime="2026-07-29 19:08:51.123549234 +0000 UTC m=+497.423221778"
Jul 29 19:09:07.442369 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.442330    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"]
Jul 29 19:09:07.449000 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.448972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.457272 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.457250    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 19:09:07.457411 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.457273    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b4wmv\""
Jul 29 19:09:07.457411 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.457300    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 19:09:07.458807 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.458788    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 19:09:07.461551 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.461530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdade87f-598a-4f31-9eb7-3dcf4ef08780-webhook-cert\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.461668 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.461566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bdade87f-598a-4f31-9eb7-3dcf4ef08780-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.461668 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.461640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bdade87f-598a-4f31-9eb7-3dcf4ef08780-manifests\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.461773 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.461678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdade87f-598a-4f31-9eb7-3dcf4ef08780-tmp\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.461773 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.461730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnhks\" (UniqueName: \"kubernetes.io/projected/bdade87f-598a-4f31-9eb7-3dcf4ef08780-kube-api-access-fnhks\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.463061 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.463042    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 19:09:07.562843 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.562804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnhks\" (UniqueName: \"kubernetes.io/projected/bdade87f-598a-4f31-9eb7-3dcf4ef08780-kube-api-access-fnhks\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.563041 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.562866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdade87f-598a-4f31-9eb7-3dcf4ef08780-webhook-cert\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.563041 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.562884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bdade87f-598a-4f31-9eb7-3dcf4ef08780-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.563041 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.562912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bdade87f-598a-4f31-9eb7-3dcf4ef08780-manifests\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.563041 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.562938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdade87f-598a-4f31-9eb7-3dcf4ef08780-tmp\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.563462 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.563433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bdade87f-598a-4f31-9eb7-3dcf4ef08780-manifests\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.565635 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.565607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdade87f-598a-4f31-9eb7-3dcf4ef08780-tmp\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.565963 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.565942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdade87f-598a-4f31-9eb7-3dcf4ef08780-webhook-cert\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.566047 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.566033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bdade87f-598a-4f31-9eb7-3dcf4ef08780-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.573700 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.573675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnhks\" (UniqueName: \"kubernetes.io/projected/bdade87f-598a-4f31-9eb7-3dcf4ef08780-kube-api-access-fnhks\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-v7trc\" (UID: \"bdade87f-598a-4f31-9eb7-3dcf4ef08780\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.594667 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.594622    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"]
Jul 29 19:09:07.760458 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.760344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:07.949920 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:07.949876    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"]
Jul 29 19:09:07.955010 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:09:07.954956    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdade87f_598a_4f31_9eb7_3dcf4ef08780.slice/crio-53b0a354957347c1ae2857cd6138c38d768a944b34a32bc3630375fe7c3997bf WatchSource:0}: Error finding container 53b0a354957347c1ae2857cd6138c38d768a944b34a32bc3630375fe7c3997bf: Status 404 returned error can't find the container with id 53b0a354957347c1ae2857cd6138c38d768a944b34a32bc3630375fe7c3997bf
Jul 29 19:09:08.167758 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:08.167724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc" event={"ID":"bdade87f-598a-4f31-9eb7-3dcf4ef08780","Type":"ContainerStarted","Data":"53b0a354957347c1ae2857cd6138c38d768a944b34a32bc3630375fe7c3997bf"}
Jul 29 19:09:11.183921 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:11.183872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc" event={"ID":"bdade87f-598a-4f31-9eb7-3dcf4ef08780","Type":"ContainerStarted","Data":"cfeafc9f7daf41f8d188dbdf73e4ef3fabe022319c66cba0659709f023a0b2d3"}
Jul 29 19:09:12.190030 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:12.189987    2579 generic.go:356] "Generic (PLEG): container finished" podID="bdade87f-598a-4f31-9eb7-3dcf4ef08780" containerID="cfeafc9f7daf41f8d188dbdf73e4ef3fabe022319c66cba0659709f023a0b2d3" exitCode=0
Jul 29 19:09:12.195259 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:12.195227    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc" event={"ID":"bdade87f-598a-4f31-9eb7-3dcf4ef08780","Type":"ContainerDied","Data":"cfeafc9f7daf41f8d188dbdf73e4ef3fabe022319c66cba0659709f023a0b2d3"}
Jul 29 19:09:13.198569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:13.198520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc" event={"ID":"bdade87f-598a-4f31-9eb7-3dcf4ef08780","Type":"ContainerStarted","Data":"72aae2876482222ed1a0be48db1277fd9b52940bc94f44813d0d23ba55369fb1"}
Jul 29 19:09:13.199085 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:13.198635    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:13.233367 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:13.233304    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc" podStartSLOduration=1.710325157 podStartE2EDuration="6.23328462s" totalImagesPullingTime="3.39056131s" totalInitContainerRuntime="1.132398153s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:09:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:09:13.23117774 +0000 UTC m=+519.530850262" watchObservedRunningTime="2026-07-29 19:09:13.23328462 +0000 UTC m=+519.532957122"
Jul 29 19:09:24.205638 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:24.205555    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-v7trc"
Jul 29 19:09:26.129539 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.129493    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"]
Jul 29 19:09:26.133001 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.132978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:26.136254 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.136232    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 19:09:26.137568 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.137548    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 19:09:26.137669 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.137607    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 19:09:26.137737 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.137674    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hp5pw\""
Jul 29 19:09:26.157868 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.157831    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"]
Jul 29 19:09:26.247090 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.247051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2732d1a-d578-42da-912d-6dfa85c53344-metrics-cert\") pod \"lws-controller-manager-79b99cb5c6-df9jj\" (UID: \"b2732d1a-d578-42da-912d-6dfa85c53344\") " pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:26.247340 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.247144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b2732d1a-d578-42da-912d-6dfa85c53344-manager-config\") pod \"lws-controller-manager-79b99cb5c6-df9jj\" (UID: \"b2732d1a-d578-42da-912d-6dfa85c53344\") " pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:26.247340 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.247165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2732d1a-d578-42da-912d-6dfa85c53344-cert\") pod \"lws-controller-manager-79b99cb5c6-df9jj\" (UID: \"b2732d1a-d578-42da-912d-6dfa85c53344\") " pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:26.247340 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.247181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmzmg\" (UniqueName: \"kubernetes.io/projected/b2732d1a-d578-42da-912d-6dfa85c53344-kube-api-access-zmzmg\") pod \"lws-controller-manager-79b99cb5c6-df9jj\" (UID: \"b2732d1a-d578-42da-912d-6dfa85c53344\") " pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:26.348651 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.348612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b2732d1a-d578-42da-912d-6dfa85c53344-manager-config\") pod \"lws-controller-manager-79b99cb5c6-df9jj\" (UID: \"b2732d1a-d578-42da-912d-6dfa85c53344\") " pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:26.348651 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.348656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2732d1a-d578-42da-912d-6dfa85c53344-cert\") pod \"lws-controller-manager-79b99cb5c6-df9jj\" (UID: \"b2732d1a-d578-42da-912d-6dfa85c53344\") " pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:26.348975 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.348687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmzmg\" (UniqueName: \"kubernetes.io/projected/b2732d1a-d578-42da-912d-6dfa85c53344-kube-api-access-zmzmg\") pod \"lws-controller-manager-79b99cb5c6-df9jj\" (UID: \"b2732d1a-d578-42da-912d-6dfa85c53344\") " pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:26.348975 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.348727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2732d1a-d578-42da-912d-6dfa85c53344-metrics-cert\") pod \"lws-controller-manager-79b99cb5c6-df9jj\" (UID: \"b2732d1a-d578-42da-912d-6dfa85c53344\") " pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:26.349624 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.349548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b2732d1a-d578-42da-912d-6dfa85c53344-manager-config\") pod \"lws-controller-manager-79b99cb5c6-df9jj\" (UID: \"b2732d1a-d578-42da-912d-6dfa85c53344\") " pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:26.351524 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.351502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2732d1a-d578-42da-912d-6dfa85c53344-metrics-cert\") pod \"lws-controller-manager-79b99cb5c6-df9jj\" (UID: \"b2732d1a-d578-42da-912d-6dfa85c53344\") " pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:26.351738 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.351715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2732d1a-d578-42da-912d-6dfa85c53344-cert\") pod \"lws-controller-manager-79b99cb5c6-df9jj\" (UID: \"b2732d1a-d578-42da-912d-6dfa85c53344\") " pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:26.359124 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.359097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmzmg\" (UniqueName: \"kubernetes.io/projected/b2732d1a-d578-42da-912d-6dfa85c53344-kube-api-access-zmzmg\") pod \"lws-controller-manager-79b99cb5c6-df9jj\" (UID: \"b2732d1a-d578-42da-912d-6dfa85c53344\") " pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:26.443337 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.443295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:26.639783 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:26.639735    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"]
Jul 29 19:09:26.646723 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:09:26.646695    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2732d1a_d578_42da_912d_6dfa85c53344.slice/crio-0eb17963701fc3aa2e4ab3d704fc013d9fea180b4eb27f0b2b7d0cb0ebe0f9bf WatchSource:0}: Error finding container 0eb17963701fc3aa2e4ab3d704fc013d9fea180b4eb27f0b2b7d0cb0ebe0f9bf: Status 404 returned error can't find the container with id 0eb17963701fc3aa2e4ab3d704fc013d9fea180b4eb27f0b2b7d0cb0ebe0f9bf
Jul 29 19:09:27.264668 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:27.264634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj" event={"ID":"b2732d1a-d578-42da-912d-6dfa85c53344","Type":"ContainerStarted","Data":"0eb17963701fc3aa2e4ab3d704fc013d9fea180b4eb27f0b2b7d0cb0ebe0f9bf"}
Jul 29 19:09:29.057291 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.057252    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6"]
Jul 29 19:09:29.060892 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.060873    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6"
Jul 29 19:09:29.071506 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.071477    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 29 19:09:29.071658 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.071477    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 19:09:29.071658 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.071515    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 29 19:09:29.081088 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.081058    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-85knt\""
Jul 29 19:09:29.086647 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.086619    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 19:09:29.112355 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.112313    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6"]
Jul 29 19:09:29.176136 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.176096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7rv9\" (UniqueName: \"kubernetes.io/projected/d7a34574-93af-49d6-a45d-066659d086ca-kube-api-access-p7rv9\") pod \"kube-auth-proxy-5f5845fd56-v72b6\" (UID: \"d7a34574-93af-49d6-a45d-066659d086ca\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6"
Jul 29 19:09:29.176136 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.176137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a34574-93af-49d6-a45d-066659d086ca-tls-certs\") pod \"kube-auth-proxy-5f5845fd56-v72b6\" (UID: \"d7a34574-93af-49d6-a45d-066659d086ca\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6"
Jul 29 19:09:29.176439 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.176157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7a34574-93af-49d6-a45d-066659d086ca-tmp\") pod \"kube-auth-proxy-5f5845fd56-v72b6\" (UID: \"d7a34574-93af-49d6-a45d-066659d086ca\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6"
Jul 29 19:09:29.276367 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.276330    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj" event={"ID":"b2732d1a-d578-42da-912d-6dfa85c53344","Type":"ContainerStarted","Data":"fc6b6565e0a5d3a56a18085b9f6202c0c3b11a745275fd3d48f4e6f0988dc8e2"}
Jul 29 19:09:29.276578 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.276457    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:09:29.276806 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.276790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7rv9\" (UniqueName: \"kubernetes.io/projected/d7a34574-93af-49d6-a45d-066659d086ca-kube-api-access-p7rv9\") pod \"kube-auth-proxy-5f5845fd56-v72b6\" (UID: \"d7a34574-93af-49d6-a45d-066659d086ca\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6"
Jul 29 19:09:29.276880 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.276823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a34574-93af-49d6-a45d-066659d086ca-tls-certs\") pod \"kube-auth-proxy-5f5845fd56-v72b6\" (UID: \"d7a34574-93af-49d6-a45d-066659d086ca\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6"
Jul 29 19:09:29.276880 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.276844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7a34574-93af-49d6-a45d-066659d086ca-tmp\") pod \"kube-auth-proxy-5f5845fd56-v72b6\" (UID: \"d7a34574-93af-49d6-a45d-066659d086ca\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6"
Jul 29 19:09:29.279355 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.279329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7a34574-93af-49d6-a45d-066659d086ca-tmp\") pod \"kube-auth-proxy-5f5845fd56-v72b6\" (UID: \"d7a34574-93af-49d6-a45d-066659d086ca\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6"
Jul 29 19:09:29.279674 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.279654    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a34574-93af-49d6-a45d-066659d086ca-tls-certs\") pod \"kube-auth-proxy-5f5845fd56-v72b6\" (UID: \"d7a34574-93af-49d6-a45d-066659d086ca\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6"
Jul 29 19:09:29.303369 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.303332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7rv9\" (UniqueName: \"kubernetes.io/projected/d7a34574-93af-49d6-a45d-066659d086ca-kube-api-access-p7rv9\") pod \"kube-auth-proxy-5f5845fd56-v72b6\" (UID: \"d7a34574-93af-49d6-a45d-066659d086ca\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6"
Jul 29 19:09:29.323101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.322982    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj" podStartSLOduration=1.4555790370000001 podStartE2EDuration="3.322961513s" totalImagesPullingTime="1.867382476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:09:29.320370436 +0000 UTC m=+535.620042940" watchObservedRunningTime="2026-07-29 19:09:29.322961513 +0000 UTC m=+535.622634013"
Jul 29 19:09:29.370713 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.370673    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6"
Jul 29 19:09:29.577374 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:29.576929    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6"]
Jul 29 19:09:29.580049 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:09:29.580014    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a34574_93af_49d6_a45d_066659d086ca.slice/crio-13ebc9785ec9c7a9f0e3e10d5286e839574992872487c389e811721dc6ee76f1 WatchSource:0}: Error finding container 13ebc9785ec9c7a9f0e3e10d5286e839574992872487c389e811721dc6ee76f1: Status 404 returned error can't find the container with id 13ebc9785ec9c7a9f0e3e10d5286e839574992872487c389e811721dc6ee76f1
Jul 29 19:09:30.283164 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:30.283116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6" event={"ID":"d7a34574-93af-49d6-a45d-066659d086ca","Type":"ContainerStarted","Data":"13ebc9785ec9c7a9f0e3e10d5286e839574992872487c389e811721dc6ee76f1"}
Jul 29 19:09:33.301575 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:33.301540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6" event={"ID":"d7a34574-93af-49d6-a45d-066659d086ca","Type":"ContainerStarted","Data":"b135907cbc4b2a6c7c593bf3c44b30025dc24dc8eabbece50773393fba72aec7"}
Jul 29 19:09:33.319260 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:33.319176    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f5845fd56-v72b6" podStartSLOduration=1.411042586 podStartE2EDuration="4.31915746s" totalImagesPullingTime="2.908114874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:09:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:09:33.318571347 +0000 UTC m=+539.618243847" watchObservedRunningTime="2026-07-29 19:09:33.31915746 +0000 UTC m=+539.618829960"
Jul 29 19:09:40.285712 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:09:40.285667    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-79b99cb5c6-df9jj"
Jul 29 19:10:34.120309 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:10:34.120278    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:10:34.123172 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:10:34.123152    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:11:13.883019 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:13.882942    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gqfvt"]
Jul 29 19:11:13.885033 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:13.885019    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gqfvt"
Jul 29 19:11:13.889173 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:13.889152    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 19:11:13.889173 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:13.889164    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 19:11:13.890092 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:13.890075    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-m52d4\""
Jul 29 19:11:13.890231 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:13.890079    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 29 19:11:13.904170 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:13.904144    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gqfvt"]
Jul 29 19:11:14.016404 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:14.016375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brql\" (UniqueName: \"kubernetes.io/projected/e3be0780-4663-4b6a-9ebd-26e7888aca18-kube-api-access-9brql\") pod \"dns-operator-controller-manager-864dd78c4c-gqfvt\" (UID: \"e3be0780-4663-4b6a-9ebd-26e7888aca18\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gqfvt"
Jul 29 19:11:14.117805 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:14.117767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brql\" (UniqueName: \"kubernetes.io/projected/e3be0780-4663-4b6a-9ebd-26e7888aca18-kube-api-access-9brql\") pod \"dns-operator-controller-manager-864dd78c4c-gqfvt\" (UID: \"e3be0780-4663-4b6a-9ebd-26e7888aca18\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gqfvt"
Jul 29 19:11:14.151313 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:14.151249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brql\" (UniqueName: \"kubernetes.io/projected/e3be0780-4663-4b6a-9ebd-26e7888aca18-kube-api-access-9brql\") pod \"dns-operator-controller-manager-864dd78c4c-gqfvt\" (UID: \"e3be0780-4663-4b6a-9ebd-26e7888aca18\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gqfvt"
Jul 29 19:11:14.195440 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:14.195407    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gqfvt"
Jul 29 19:11:14.342230 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:14.342204    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gqfvt"]
Jul 29 19:11:14.343629 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:11:14.343604    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3be0780_4663_4b6a_9ebd_26e7888aca18.slice/crio-7e55333f099b5d17868a078f75a4ee7f8c2abeba8c6e6bf5028dab7c965a02d5 WatchSource:0}: Error finding container 7e55333f099b5d17868a078f75a4ee7f8c2abeba8c6e6bf5028dab7c965a02d5: Status 404 returned error can't find the container with id 7e55333f099b5d17868a078f75a4ee7f8c2abeba8c6e6bf5028dab7c965a02d5
Jul 29 19:11:14.345642 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:14.345624    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:11:14.758908 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:14.758877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gqfvt" event={"ID":"e3be0780-4663-4b6a-9ebd-26e7888aca18","Type":"ContainerStarted","Data":"7e55333f099b5d17868a078f75a4ee7f8c2abeba8c6e6bf5028dab7c965a02d5"}
Jul 29 19:11:16.770109 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:16.770071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gqfvt" event={"ID":"e3be0780-4663-4b6a-9ebd-26e7888aca18","Type":"ContainerStarted","Data":"2a2ab144c6429f28cc1f393941e658f9f33183b39c618b31530ac466b55cda94"}
Jul 29 19:11:16.770604 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:16.770179    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gqfvt"
Jul 29 19:11:16.791653 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:16.791458    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gqfvt" podStartSLOduration=1.521712733 podStartE2EDuration="3.791439218s" totalImagesPullingTime="2.269726485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:16.79112192 +0000 UTC m=+643.090794423" watchObservedRunningTime="2026-07-29 19:11:16.791439218 +0000 UTC m=+643.091111752"
Jul 29 19:11:27.776103 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:27.776070    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gqfvt"
Jul 29 19:11:32.284630 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:32.284599    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cgkmb"]
Jul 29 19:11:32.286852 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:32.286834    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cgkmb"
Jul 29 19:11:32.291151 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:32.291130    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8zr2p\""
Jul 29 19:11:32.309401 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:32.309377    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cgkmb"]
Jul 29 19:11:32.382588 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:32.382555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhmtk\" (UniqueName: \"kubernetes.io/projected/5b8105b3-6fcf-431d-b5a0-a7b224b58387-kube-api-access-qhmtk\") pod \"limitador-operator-controller-manager-859d7b78bd-cgkmb\" (UID: \"5b8105b3-6fcf-431d-b5a0-a7b224b58387\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cgkmb"
Jul 29 19:11:32.483809 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:32.483774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhmtk\" (UniqueName: \"kubernetes.io/projected/5b8105b3-6fcf-431d-b5a0-a7b224b58387-kube-api-access-qhmtk\") pod \"limitador-operator-controller-manager-859d7b78bd-cgkmb\" (UID: \"5b8105b3-6fcf-431d-b5a0-a7b224b58387\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cgkmb"
Jul 29 19:11:32.501538 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:32.501508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhmtk\" (UniqueName: \"kubernetes.io/projected/5b8105b3-6fcf-431d-b5a0-a7b224b58387-kube-api-access-qhmtk\") pod \"limitador-operator-controller-manager-859d7b78bd-cgkmb\" (UID: \"5b8105b3-6fcf-431d-b5a0-a7b224b58387\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cgkmb"
Jul 29 19:11:32.597369 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:32.597283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cgkmb"
Jul 29 19:11:32.728355 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:32.728216    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cgkmb"]
Jul 29 19:11:32.731256 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:11:32.731229    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8105b3_6fcf_431d_b5a0_a7b224b58387.slice/crio-bc98fb28cd3dbcaab06fffe5b439677f30bbc13f9f9e8f963aadbab408caf8e6 WatchSource:0}: Error finding container bc98fb28cd3dbcaab06fffe5b439677f30bbc13f9f9e8f963aadbab408caf8e6: Status 404 returned error can't find the container with id bc98fb28cd3dbcaab06fffe5b439677f30bbc13f9f9e8f963aadbab408caf8e6
Jul 29 19:11:32.839814 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:32.839784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cgkmb" event={"ID":"5b8105b3-6fcf-431d-b5a0-a7b224b58387","Type":"ContainerStarted","Data":"bc98fb28cd3dbcaab06fffe5b439677f30bbc13f9f9e8f963aadbab408caf8e6"}
Jul 29 19:11:34.852852 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:34.852803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cgkmb" event={"ID":"5b8105b3-6fcf-431d-b5a0-a7b224b58387","Type":"ContainerStarted","Data":"73cd0a3f4b2b57b4379c134de28a6020dc3be36e06c9ea26483834d3ff973e81"}
Jul 29 19:11:34.853351 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:34.852940    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cgkmb"
Jul 29 19:11:34.875559 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:34.875510    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cgkmb" podStartSLOduration=1.286229279 podStartE2EDuration="2.875494235s" totalImagesPullingTime="1.589264956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:34.873939394 +0000 UTC m=+661.173611935" watchObservedRunningTime="2026-07-29 19:11:34.875494235 +0000 UTC m=+661.175166734"
Jul 29 19:11:45.860017 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:11:45.859986    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cgkmb"
Jul 29 19:12:30.199136 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:30.199060    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-d2qms"]
Jul 29 19:12:30.202597 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:30.202575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-d2qms"
Jul 29 19:12:30.205441 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:30.205421    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qllfn\""
Jul 29 19:12:30.214213 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:30.213911    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-d2qms"]
Jul 29 19:12:30.318072 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:30.318038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42hwc\" (UniqueName: \"kubernetes.io/projected/fe695842-52b8-4c14-983d-81937b08152c-kube-api-access-42hwc\") pod \"authorino-58cd955675-d2qms\" (UID: \"fe695842-52b8-4c14-983d-81937b08152c\") " pod="kuadrant-system/authorino-58cd955675-d2qms"
Jul 29 19:12:30.419135 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:30.419092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42hwc\" (UniqueName: \"kubernetes.io/projected/fe695842-52b8-4c14-983d-81937b08152c-kube-api-access-42hwc\") pod \"authorino-58cd955675-d2qms\" (UID: \"fe695842-52b8-4c14-983d-81937b08152c\") " pod="kuadrant-system/authorino-58cd955675-d2qms"
Jul 29 19:12:30.428286 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:30.428256    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42hwc\" (UniqueName: \"kubernetes.io/projected/fe695842-52b8-4c14-983d-81937b08152c-kube-api-access-42hwc\") pod \"authorino-58cd955675-d2qms\" (UID: \"fe695842-52b8-4c14-983d-81937b08152c\") " pod="kuadrant-system/authorino-58cd955675-d2qms"
Jul 29 19:12:30.518971 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:30.518889    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-d2qms"
Jul 29 19:12:30.656329 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:30.656292    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-d2qms"]
Jul 29 19:12:30.659568 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:12:30.659539    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe695842_52b8_4c14_983d_81937b08152c.slice/crio-c2b76581895049bcdd0b7c350c3d04c43c3b12e20a8f6b410ff2c0cf6f0fe2ae WatchSource:0}: Error finding container c2b76581895049bcdd0b7c350c3d04c43c3b12e20a8f6b410ff2c0cf6f0fe2ae: Status 404 returned error can't find the container with id c2b76581895049bcdd0b7c350c3d04c43c3b12e20a8f6b410ff2c0cf6f0fe2ae
Jul 29 19:12:31.111013 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:31.110972    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-d2qms" event={"ID":"fe695842-52b8-4c14-983d-81937b08152c","Type":"ContainerStarted","Data":"c2b76581895049bcdd0b7c350c3d04c43c3b12e20a8f6b410ff2c0cf6f0fe2ae"}
Jul 29 19:12:34.130240 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:34.130186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-d2qms" event={"ID":"fe695842-52b8-4c14-983d-81937b08152c","Type":"ContainerStarted","Data":"b18a7c7e98bd082e675e5dd5197c63a8acd0f04a2da359b96d88fb833a7ef975"}
Jul 29 19:12:34.149118 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:34.149059    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-d2qms" podStartSLOduration=1.016930805 podStartE2EDuration="4.149042385s" totalImagesPullingTime="3.13211158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:12:34.148343691 +0000 UTC m=+720.448016193" watchObservedRunningTime="2026-07-29 19:12:34.149042385 +0000 UTC m=+720.448714886"
Jul 29 19:12:35.628745 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:35.628703    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-d2qms"]
Jul 29 19:12:36.139289 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:36.139245    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-d2qms" podUID="fe695842-52b8-4c14-983d-81937b08152c" containerName="authorino" containerID="cri-o://b18a7c7e98bd082e675e5dd5197c63a8acd0f04a2da359b96d88fb833a7ef975" gracePeriod=30
Jul 29 19:12:36.384270 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:36.384245    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-d2qms"
Jul 29 19:12:36.477384 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:36.477301    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42hwc\" (UniqueName: \"kubernetes.io/projected/fe695842-52b8-4c14-983d-81937b08152c-kube-api-access-42hwc\") pod \"fe695842-52b8-4c14-983d-81937b08152c\" (UID: \"fe695842-52b8-4c14-983d-81937b08152c\") "
Jul 29 19:12:36.479671 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:36.479647    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe695842-52b8-4c14-983d-81937b08152c-kube-api-access-42hwc" (OuterVolumeSpecName: "kube-api-access-42hwc") pod "fe695842-52b8-4c14-983d-81937b08152c" (UID: "fe695842-52b8-4c14-983d-81937b08152c"). InnerVolumeSpecName "kube-api-access-42hwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:12:36.579060 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:36.579021    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42hwc\" (UniqueName: \"kubernetes.io/projected/fe695842-52b8-4c14-983d-81937b08152c-kube-api-access-42hwc\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:12:37.145148 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:37.145109    2579 generic.go:356] "Generic (PLEG): container finished" podID="fe695842-52b8-4c14-983d-81937b08152c" containerID="b18a7c7e98bd082e675e5dd5197c63a8acd0f04a2da359b96d88fb833a7ef975" exitCode=0
Jul 29 19:12:37.145148 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:37.145152    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-d2qms"
Jul 29 19:12:37.145643 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:37.145165    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-d2qms" event={"ID":"fe695842-52b8-4c14-983d-81937b08152c","Type":"ContainerDied","Data":"b18a7c7e98bd082e675e5dd5197c63a8acd0f04a2da359b96d88fb833a7ef975"}
Jul 29 19:12:37.145643 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:37.145232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-d2qms" event={"ID":"fe695842-52b8-4c14-983d-81937b08152c","Type":"ContainerDied","Data":"c2b76581895049bcdd0b7c350c3d04c43c3b12e20a8f6b410ff2c0cf6f0fe2ae"}
Jul 29 19:12:37.145643 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:37.145255    2579 scope.go:117] "RemoveContainer" containerID="b18a7c7e98bd082e675e5dd5197c63a8acd0f04a2da359b96d88fb833a7ef975"
Jul 29 19:12:37.154316 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:37.154299    2579 scope.go:117] "RemoveContainer" containerID="b18a7c7e98bd082e675e5dd5197c63a8acd0f04a2da359b96d88fb833a7ef975"
Jul 29 19:12:37.154604 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:12:37.154586    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18a7c7e98bd082e675e5dd5197c63a8acd0f04a2da359b96d88fb833a7ef975\": container with ID starting with b18a7c7e98bd082e675e5dd5197c63a8acd0f04a2da359b96d88fb833a7ef975 not found: ID does not exist" containerID="b18a7c7e98bd082e675e5dd5197c63a8acd0f04a2da359b96d88fb833a7ef975"
Jul 29 19:12:37.154662 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:37.154615    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18a7c7e98bd082e675e5dd5197c63a8acd0f04a2da359b96d88fb833a7ef975"} err="failed to get container status \"b18a7c7e98bd082e675e5dd5197c63a8acd0f04a2da359b96d88fb833a7ef975\": rpc error: code = NotFound desc = could not find container \"b18a7c7e98bd082e675e5dd5197c63a8acd0f04a2da359b96d88fb833a7ef975\": container with ID starting with b18a7c7e98bd082e675e5dd5197c63a8acd0f04a2da359b96d88fb833a7ef975 not found: ID does not exist"
Jul 29 19:12:37.166867 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:37.166834    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-d2qms"]
Jul 29 19:12:37.170856 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:37.170830    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-d2qms"]
Jul 29 19:12:38.195801 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:12:38.195757    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe695842-52b8-4c14-983d-81937b08152c" path="/var/lib/kubelet/pods/fe695842-52b8-4c14-983d-81937b08152c/volumes"
Jul 29 19:13:00.239708 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.239677    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6w8f8"]
Jul 29 19:13:00.240171 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.240036    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe695842-52b8-4c14-983d-81937b08152c" containerName="authorino"
Jul 29 19:13:00.240171 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.240047    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe695842-52b8-4c14-983d-81937b08152c" containerName="authorino"
Jul 29 19:13:00.240171 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.240110    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe695842-52b8-4c14-983d-81937b08152c" containerName="authorino"
Jul 29 19:13:00.243045 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.243025    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6w8f8"
Jul 29 19:13:00.245970 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.245946    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qllfn\""
Jul 29 19:13:00.254335 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.254301    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6w8f8"]
Jul 29 19:13:00.301376 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.301343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8q6\" (UniqueName: \"kubernetes.io/projected/e3c16153-f453-4d66-a9d6-8e158c11e897-kube-api-access-lj8q6\") pod \"authorino-5d4b67c446-6w8f8\" (UID: \"e3c16153-f453-4d66-a9d6-8e158c11e897\") " pod="kuadrant-system/authorino-5d4b67c446-6w8f8"
Jul 29 19:13:00.402867 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.402830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8q6\" (UniqueName: \"kubernetes.io/projected/e3c16153-f453-4d66-a9d6-8e158c11e897-kube-api-access-lj8q6\") pod \"authorino-5d4b67c446-6w8f8\" (UID: \"e3c16153-f453-4d66-a9d6-8e158c11e897\") " pod="kuadrant-system/authorino-5d4b67c446-6w8f8"
Jul 29 19:13:00.414363 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.414327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8q6\" (UniqueName: \"kubernetes.io/projected/e3c16153-f453-4d66-a9d6-8e158c11e897-kube-api-access-lj8q6\") pod \"authorino-5d4b67c446-6w8f8\" (UID: \"e3c16153-f453-4d66-a9d6-8e158c11e897\") " pod="kuadrant-system/authorino-5d4b67c446-6w8f8"
Jul 29 19:13:00.434808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.434774    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6w8f8"]
Jul 29 19:13:00.435076 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.435059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6w8f8"
Jul 29 19:13:00.548004 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.547972    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7475ccd557-46825"]
Jul 29 19:13:00.555117 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.555093    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7475ccd557-46825"
Jul 29 19:13:00.558401 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.558379    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 29 19:13:00.559608 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.559578    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7475ccd557-46825"]
Jul 29 19:13:00.574419 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.574398    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6w8f8"]
Jul 29 19:13:00.576833 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:13:00.576804    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3c16153_f453_4d66_a9d6_8e158c11e897.slice/crio-0718580e30668ee6c26eff451099e0704569cbc03a478e1b5b9ad12418289829 WatchSource:0}: Error finding container 0718580e30668ee6c26eff451099e0704569cbc03a478e1b5b9ad12418289829: Status 404 returned error can't find the container with id 0718580e30668ee6c26eff451099e0704569cbc03a478e1b5b9ad12418289829
Jul 29 19:13:00.705037 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.704999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxwc8\" (UniqueName: \"kubernetes.io/projected/ba00e9c1-80cf-418a-9467-91c8c18a41aa-kube-api-access-fxwc8\") pod \"authorino-7475ccd557-46825\" (UID: \"ba00e9c1-80cf-418a-9467-91c8c18a41aa\") " pod="kuadrant-system/authorino-7475ccd557-46825"
Jul 29 19:13:00.705249 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.705046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba00e9c1-80cf-418a-9467-91c8c18a41aa-tls-cert\") pod \"authorino-7475ccd557-46825\" (UID: \"ba00e9c1-80cf-418a-9467-91c8c18a41aa\") " pod="kuadrant-system/authorino-7475ccd557-46825"
Jul 29 19:13:00.806811 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.806118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxwc8\" (UniqueName: \"kubernetes.io/projected/ba00e9c1-80cf-418a-9467-91c8c18a41aa-kube-api-access-fxwc8\") pod \"authorino-7475ccd557-46825\" (UID: \"ba00e9c1-80cf-418a-9467-91c8c18a41aa\") " pod="kuadrant-system/authorino-7475ccd557-46825"
Jul 29 19:13:00.806811 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.806178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba00e9c1-80cf-418a-9467-91c8c18a41aa-tls-cert\") pod \"authorino-7475ccd557-46825\" (UID: \"ba00e9c1-80cf-418a-9467-91c8c18a41aa\") " pod="kuadrant-system/authorino-7475ccd557-46825"
Jul 29 19:13:00.810435 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.810407    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba00e9c1-80cf-418a-9467-91c8c18a41aa-tls-cert\") pod \"authorino-7475ccd557-46825\" (UID: \"ba00e9c1-80cf-418a-9467-91c8c18a41aa\") " pod="kuadrant-system/authorino-7475ccd557-46825"
Jul 29 19:13:00.815597 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.815576    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxwc8\" (UniqueName: \"kubernetes.io/projected/ba00e9c1-80cf-418a-9467-91c8c18a41aa-kube-api-access-fxwc8\") pod \"authorino-7475ccd557-46825\" (UID: \"ba00e9c1-80cf-418a-9467-91c8c18a41aa\") " pod="kuadrant-system/authorino-7475ccd557-46825"
Jul 29 19:13:00.866622 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.866584    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7475ccd557-46825"
Jul 29 19:13:00.997110 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:00.997085    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7475ccd557-46825"]
Jul 29 19:13:00.999273 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:13:00.999240    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba00e9c1_80cf_418a_9467_91c8c18a41aa.slice/crio-72efee6a941255c4913399d1b20ffecdf7ba7112edc4ff17a7ced1e207534f7b WatchSource:0}: Error finding container 72efee6a941255c4913399d1b20ffecdf7ba7112edc4ff17a7ced1e207534f7b: Status 404 returned error can't find the container with id 72efee6a941255c4913399d1b20ffecdf7ba7112edc4ff17a7ced1e207534f7b
Jul 29 19:13:01.255269 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:01.255226    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6w8f8" event={"ID":"e3c16153-f453-4d66-a9d6-8e158c11e897","Type":"ContainerStarted","Data":"c8879f3d114bb2203c13c0b853d919db7cd4be2c51f360e6c294817f58f732e0"}
Jul 29 19:13:01.255269 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:01.255258    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-6w8f8" podUID="e3c16153-f453-4d66-a9d6-8e158c11e897" containerName="authorino" containerID="cri-o://c8879f3d114bb2203c13c0b853d919db7cd4be2c51f360e6c294817f58f732e0" gracePeriod=30
Jul 29 19:13:01.255788 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:01.255275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6w8f8" event={"ID":"e3c16153-f453-4d66-a9d6-8e158c11e897","Type":"ContainerStarted","Data":"0718580e30668ee6c26eff451099e0704569cbc03a478e1b5b9ad12418289829"}
Jul 29 19:13:01.256601 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:01.256574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7475ccd557-46825" event={"ID":"ba00e9c1-80cf-418a-9467-91c8c18a41aa","Type":"ContainerStarted","Data":"72efee6a941255c4913399d1b20ffecdf7ba7112edc4ff17a7ced1e207534f7b"}
Jul 29 19:13:01.271520 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:01.271472    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-6w8f8" podStartSLOduration=0.879230565 podStartE2EDuration="1.271454421s" totalImagesPullingTime="392.223856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:01.270889622 +0000 UTC m=+747.570562144" watchObservedRunningTime="2026-07-29 19:13:01.271454421 +0000 UTC m=+747.571126923"
Jul 29 19:13:01.518962 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:01.518890    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6w8f8"
Jul 29 19:13:01.612258 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:01.612221    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj8q6\" (UniqueName: \"kubernetes.io/projected/e3c16153-f453-4d66-a9d6-8e158c11e897-kube-api-access-lj8q6\") pod \"e3c16153-f453-4d66-a9d6-8e158c11e897\" (UID: \"e3c16153-f453-4d66-a9d6-8e158c11e897\") "
Jul 29 19:13:01.614497 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:01.614465    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c16153-f453-4d66-a9d6-8e158c11e897-kube-api-access-lj8q6" (OuterVolumeSpecName: "kube-api-access-lj8q6") pod "e3c16153-f453-4d66-a9d6-8e158c11e897" (UID: "e3c16153-f453-4d66-a9d6-8e158c11e897"). InnerVolumeSpecName "kube-api-access-lj8q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:13:01.713739 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:01.713691    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj8q6\" (UniqueName: \"kubernetes.io/projected/e3c16153-f453-4d66-a9d6-8e158c11e897-kube-api-access-lj8q6\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:13:02.262420 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:02.262376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7475ccd557-46825" event={"ID":"ba00e9c1-80cf-418a-9467-91c8c18a41aa","Type":"ContainerStarted","Data":"70b1bc1ad6ed767e9a91491a347dabd8ead29956c9f259a18895293d29804139"}
Jul 29 19:13:02.263619 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:02.263596    2579 generic.go:356] "Generic (PLEG): container finished" podID="e3c16153-f453-4d66-a9d6-8e158c11e897" containerID="c8879f3d114bb2203c13c0b853d919db7cd4be2c51f360e6c294817f58f732e0" exitCode=0
Jul 29 19:13:02.263721 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:02.263631    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6w8f8"
Jul 29 19:13:02.263721 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:02.263656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6w8f8" event={"ID":"e3c16153-f453-4d66-a9d6-8e158c11e897","Type":"ContainerDied","Data":"c8879f3d114bb2203c13c0b853d919db7cd4be2c51f360e6c294817f58f732e0"}
Jul 29 19:13:02.263721 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:02.263683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6w8f8" event={"ID":"e3c16153-f453-4d66-a9d6-8e158c11e897","Type":"ContainerDied","Data":"0718580e30668ee6c26eff451099e0704569cbc03a478e1b5b9ad12418289829"}
Jul 29 19:13:02.263834 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:02.263699    2579 scope.go:117] "RemoveContainer" containerID="c8879f3d114bb2203c13c0b853d919db7cd4be2c51f360e6c294817f58f732e0"
Jul 29 19:13:02.272143 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:02.272127    2579 scope.go:117] "RemoveContainer" containerID="c8879f3d114bb2203c13c0b853d919db7cd4be2c51f360e6c294817f58f732e0"
Jul 29 19:13:02.272458 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:13:02.272439    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8879f3d114bb2203c13c0b853d919db7cd4be2c51f360e6c294817f58f732e0\": container with ID starting with c8879f3d114bb2203c13c0b853d919db7cd4be2c51f360e6c294817f58f732e0 not found: ID does not exist" containerID="c8879f3d114bb2203c13c0b853d919db7cd4be2c51f360e6c294817f58f732e0"
Jul 29 19:13:02.272521 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:02.272468    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8879f3d114bb2203c13c0b853d919db7cd4be2c51f360e6c294817f58f732e0"} err="failed to get container status \"c8879f3d114bb2203c13c0b853d919db7cd4be2c51f360e6c294817f58f732e0\": rpc error: code = NotFound desc = could not find container \"c8879f3d114bb2203c13c0b853d919db7cd4be2c51f360e6c294817f58f732e0\": container with ID starting with c8879f3d114bb2203c13c0b853d919db7cd4be2c51f360e6c294817f58f732e0 not found: ID does not exist"
Jul 29 19:13:02.281797 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:02.281755    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7475ccd557-46825" podStartSLOduration=1.876257925 podStartE2EDuration="2.281740175s" totalImagesPullingTime="405.48225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:02.280611371 +0000 UTC m=+748.580283870" watchObservedRunningTime="2026-07-29 19:13:02.281740175 +0000 UTC m=+748.581412728"
Jul 29 19:13:02.298348 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:02.298312    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6w8f8"]
Jul 29 19:13:02.303939 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:02.303910    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6w8f8"]
Jul 29 19:13:04.196054 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:04.196016    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3c16153-f453-4d66-a9d6-8e158c11e897" path="/var/lib/kubelet/pods/e3c16153-f453-4d66-a9d6-8e158c11e897/volumes"
Jul 29 19:13:44.150093 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.150061    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-dhqmz"]
Jul 29 19:13:44.150546 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.150472    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3c16153-f453-4d66-a9d6-8e158c11e897" containerName="authorino"
Jul 29 19:13:44.150546 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.150485    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3c16153-f453-4d66-a9d6-8e158c11e897" containerName="authorino"
Jul 29 19:13:44.150622 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.150555    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3c16153-f453-4d66-a9d6-8e158c11e897" containerName="authorino"
Jul 29 19:13:44.155857 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.155827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-dhqmz"
Jul 29 19:13:44.159450 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.159418    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 29 19:13:44.159450 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.159420    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gp7r8\""
Jul 29 19:13:44.163505 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.163472    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-dhqmz"]
Jul 29 19:13:44.284026 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.283996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af11b487-3ae7-4c80-b7d1-1a54e4593381-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-dhqmz\" (UID: \"af11b487-3ae7-4c80-b7d1-1a54e4593381\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dhqmz"
Jul 29 19:13:44.284250 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.284039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt6sl\" (UniqueName: \"kubernetes.io/projected/af11b487-3ae7-4c80-b7d1-1a54e4593381-kube-api-access-vt6sl\") pod \"payload-processing-5cc6cd444b-dhqmz\" (UID: \"af11b487-3ae7-4c80-b7d1-1a54e4593381\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dhqmz"
Jul 29 19:13:44.385098 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.385063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af11b487-3ae7-4c80-b7d1-1a54e4593381-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-dhqmz\" (UID: \"af11b487-3ae7-4c80-b7d1-1a54e4593381\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dhqmz"
Jul 29 19:13:44.385098 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.385107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6sl\" (UniqueName: \"kubernetes.io/projected/af11b487-3ae7-4c80-b7d1-1a54e4593381-kube-api-access-vt6sl\") pod \"payload-processing-5cc6cd444b-dhqmz\" (UID: \"af11b487-3ae7-4c80-b7d1-1a54e4593381\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dhqmz"
Jul 29 19:13:44.385873 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.385848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af11b487-3ae7-4c80-b7d1-1a54e4593381-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-dhqmz\" (UID: \"af11b487-3ae7-4c80-b7d1-1a54e4593381\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dhqmz"
Jul 29 19:13:44.393425 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.393400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6sl\" (UniqueName: \"kubernetes.io/projected/af11b487-3ae7-4c80-b7d1-1a54e4593381-kube-api-access-vt6sl\") pod \"payload-processing-5cc6cd444b-dhqmz\" (UID: \"af11b487-3ae7-4c80-b7d1-1a54e4593381\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dhqmz"
Jul 29 19:13:44.469458 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.469362    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-dhqmz"
Jul 29 19:13:44.810182 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:44.810154    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-dhqmz"]
Jul 29 19:13:44.811635 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:13:44.811608    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf11b487_3ae7_4c80_b7d1_1a54e4593381.slice/crio-367c37a5f84276c1370b11f3f21e576957a046e6b81702c8c68a574f1c7594d4 WatchSource:0}: Error finding container 367c37a5f84276c1370b11f3f21e576957a046e6b81702c8c68a574f1c7594d4: Status 404 returned error can't find the container with id 367c37a5f84276c1370b11f3f21e576957a046e6b81702c8c68a574f1c7594d4
Jul 29 19:13:45.478284 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:45.478248    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-dhqmz" event={"ID":"af11b487-3ae7-4c80-b7d1-1a54e4593381","Type":"ContainerStarted","Data":"367c37a5f84276c1370b11f3f21e576957a046e6b81702c8c68a574f1c7594d4"}
Jul 29 19:13:47.492794 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:47.492693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-dhqmz" event={"ID":"af11b487-3ae7-4c80-b7d1-1a54e4593381","Type":"ContainerStarted","Data":"105c8ed504a5d82b36713f3f9fe8e8785b5fe3f61979836e3978a3477e0e6d79"}
Jul 29 19:13:47.493163 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:47.492892    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-dhqmz"
Jul 29 19:13:47.513615 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:47.513017    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-dhqmz" podStartSLOduration=1.155113466 podStartE2EDuration="3.512996918s" totalImagesPullingTime="2.357883452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:47.510130488 +0000 UTC m=+793.809802992" watchObservedRunningTime="2026-07-29 19:13:47.512996918 +0000 UTC m=+793.812669419"
Jul 29 19:13:58.500437 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:13:58.500360    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-dhqmz"
Jul 29 19:14:21.699641 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.696881    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"]
Jul 29 19:14:21.706301 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.706276    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.710313 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.710274    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 19:14:21.711048 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.711017    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 19:14:21.711048 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.711041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lj9bm\""
Jul 29 19:14:21.711335 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.711320    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 29 19:14:21.713248 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.713225    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"]
Jul 29 19:14:21.819021 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.818983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f203259-5677-4110-8f89-9719877d36fb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.819253 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.819041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f203259-5677-4110-8f89-9719877d36fb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.819253 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.819081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f203259-5677-4110-8f89-9719877d36fb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.819253 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.819122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f203259-5677-4110-8f89-9719877d36fb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.819253 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.819160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f203259-5677-4110-8f89-9719877d36fb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.819440 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.819275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vv57\" (UniqueName: \"kubernetes.io/projected/2f203259-5677-4110-8f89-9719877d36fb-kube-api-access-8vv57\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.919939 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.919902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f203259-5677-4110-8f89-9719877d36fb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.920105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.919966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vv57\" (UniqueName: \"kubernetes.io/projected/2f203259-5677-4110-8f89-9719877d36fb-kube-api-access-8vv57\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.920105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.920001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f203259-5677-4110-8f89-9719877d36fb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.920105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.920032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f203259-5677-4110-8f89-9719877d36fb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.920105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.920055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f203259-5677-4110-8f89-9719877d36fb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.920331 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.920147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f203259-5677-4110-8f89-9719877d36fb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.920382 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.920368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f203259-5677-4110-8f89-9719877d36fb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.920492 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.920473    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f203259-5677-4110-8f89-9719877d36fb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.920558 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.920513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f203259-5677-4110-8f89-9719877d36fb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.922406 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.922386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f203259-5677-4110-8f89-9719877d36fb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.922764 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.922745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f203259-5677-4110-8f89-9719877d36fb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:21.930056 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:21.930031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vv57\" (UniqueName: \"kubernetes.io/projected/2f203259-5677-4110-8f89-9719877d36fb-kube-api-access-8vv57\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dgfkb\" (UID: \"2f203259-5677-4110-8f89-9719877d36fb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:22.019424 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:22.019340    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:22.154955 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:22.154927    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"]
Jul 29 19:14:22.157078 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:14:22.157056    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f203259_5677_4110_8f89_9719877d36fb.slice/crio-8ca0d99c073ba2a2c3b8cc36d73a23184cb2313c6644bb171e1d9783ce57e8a1 WatchSource:0}: Error finding container 8ca0d99c073ba2a2c3b8cc36d73a23184cb2313c6644bb171e1d9783ce57e8a1: Status 404 returned error can't find the container with id 8ca0d99c073ba2a2c3b8cc36d73a23184cb2313c6644bb171e1d9783ce57e8a1
Jul 29 19:14:22.660282 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:22.660246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb" event={"ID":"2f203259-5677-4110-8f89-9719877d36fb","Type":"ContainerStarted","Data":"8ca0d99c073ba2a2c3b8cc36d73a23184cb2313c6644bb171e1d9783ce57e8a1"}
Jul 29 19:14:23.666310 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:23.666223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb" event={"ID":"2f203259-5677-4110-8f89-9719877d36fb","Type":"ContainerStarted","Data":"fd888b81c0e3591045af8a76b916265fad0e59dd9db4fff49e9ecac681d0d1a5"}
Jul 29 19:14:23.685681 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:23.685629    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb" podStartSLOduration=1.456119602 podStartE2EDuration="2.685612281s" totalImagesPullingTime="1.229492679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:14:23.683464126 +0000 UTC m=+829.983136627" watchObservedRunningTime="2026-07-29 19:14:23.685612281 +0000 UTC m=+829.985284781"
Jul 29 19:14:32.020266 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:32.020222    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:32.020748 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:32.020300    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:32.032970 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:32.032948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:14:32.720230 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:14:32.720175    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dgfkb"
Jul 29 19:15:00.136921 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:00.136888    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755875-7zvhh"]
Jul 29 19:15:00.139340 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:00.139325    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-7zvhh"
Jul 29 19:15:00.142141 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:00.142107    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq9q2\""
Jul 29 19:15:00.147004 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:00.146981    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755875-7zvhh"]
Jul 29 19:15:00.150750 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:00.150729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-7zvhh"
Jul 29 19:15:00.287632 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:00.287605    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755875-7zvhh"]
Jul 29 19:15:00.290323 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:15:00.290288    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0595a037_c243_4a25_80fb_7502bb651582.slice/crio-9bb121605745edd8a61ca1067736bfd263e0c6ccaca273aac61232acefbc398d WatchSource:0}: Error finding container 9bb121605745edd8a61ca1067736bfd263e0c6ccaca273aac61232acefbc398d: Status 404 returned error can't find the container with id 9bb121605745edd8a61ca1067736bfd263e0c6ccaca273aac61232acefbc398d
Jul 29 19:15:00.841555 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:00.841525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-7zvhh" event={"ID":"0595a037-c243-4a25-80fb-7502bb651582","Type":"ContainerStarted","Data":"9bb121605745edd8a61ca1067736bfd263e0c6ccaca273aac61232acefbc398d"}
Jul 29 19:15:04.863767 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:04.863734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-7zvhh" event={"ID":"0595a037-c243-4a25-80fb-7502bb651582","Type":"ContainerStarted","Data":"065ad2938e6287d1f5fcaf270a4d9aff47e285887ef4f50b457247342d365310"}
Jul 29 19:15:04.880447 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:04.880396    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-7zvhh" podStartSLOduration=1.213262996 podStartE2EDuration="4.880380099s" totalImagesPullingTime="3.667117103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:15:04.8790504 +0000 UTC m=+871.178722901" watchObservedRunningTime="2026-07-29 19:15:04.880380099 +0000 UTC m=+871.180052600"
Jul 29 19:15:05.869861 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:05.869824    2579 generic.go:356] "Generic (PLEG): container finished" podID="0595a037-c243-4a25-80fb-7502bb651582" containerID="065ad2938e6287d1f5fcaf270a4d9aff47e285887ef4f50b457247342d365310" exitCode=0
Jul 29 19:15:05.870272 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:05.869892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-7zvhh" event={"ID":"0595a037-c243-4a25-80fb-7502bb651582","Type":"ContainerDied","Data":"065ad2938e6287d1f5fcaf270a4d9aff47e285887ef4f50b457247342d365310"}
Jul 29 19:15:07.027484 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:07.027458    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-7zvhh"
Jul 29 19:15:07.882319 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:07.882284    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-7zvhh" event={"ID":"0595a037-c243-4a25-80fb-7502bb651582","Type":"ContainerDied","Data":"9bb121605745edd8a61ca1067736bfd263e0c6ccaca273aac61232acefbc398d"}
Jul 29 19:15:07.882319 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:07.882301    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755875-7zvhh"
Jul 29 19:15:07.882319 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:07.882330    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bb121605745edd8a61ca1067736bfd263e0c6ccaca273aac61232acefbc398d"
Jul 29 19:15:10.496378 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.496348    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"]
Jul 29 19:15:10.496762 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.496704    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0595a037-c243-4a25-80fb-7502bb651582" containerName="cleanup"
Jul 29 19:15:10.496762 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.496716    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0595a037-c243-4a25-80fb-7502bb651582" containerName="cleanup"
Jul 29 19:15:10.496836 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.496794    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0595a037-c243-4a25-80fb-7502bb651582" containerName="cleanup"
Jul 29 19:15:10.499555 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.499540    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.502472 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.502451    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 29 19:15:10.511824 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.511804    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"]
Jul 29 19:15:10.564044 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.564012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998df865-bd61-46c1-a57a-de1ecc627326-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.564044 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.564044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998df865-bd61-46c1-a57a-de1ecc627326-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.564289 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.564087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998df865-bd61-46c1-a57a-de1ecc627326-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.564289 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.564128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998df865-bd61-46c1-a57a-de1ecc627326-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.564289 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.564174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltqzs\" (UniqueName: \"kubernetes.io/projected/998df865-bd61-46c1-a57a-de1ecc627326-kube-api-access-ltqzs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.564289 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.564209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998df865-bd61-46c1-a57a-de1ecc627326-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.665371 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.665334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998df865-bd61-46c1-a57a-de1ecc627326-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.665371 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.665367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998df865-bd61-46c1-a57a-de1ecc627326-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.665570 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.665505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998df865-bd61-46c1-a57a-de1ecc627326-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.665608 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.665566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998df865-bd61-46c1-a57a-de1ecc627326-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.665641 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.665628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltqzs\" (UniqueName: \"kubernetes.io/projected/998df865-bd61-46c1-a57a-de1ecc627326-kube-api-access-ltqzs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.665671 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.665657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998df865-bd61-46c1-a57a-de1ecc627326-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.665731 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.665713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998df865-bd61-46c1-a57a-de1ecc627326-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.665916 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.665897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998df865-bd61-46c1-a57a-de1ecc627326-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.666040 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.666023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998df865-bd61-46c1-a57a-de1ecc627326-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.667878 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.667856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998df865-bd61-46c1-a57a-de1ecc627326-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.667996 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.667981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998df865-bd61-46c1-a57a-de1ecc627326-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.674866 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.674839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltqzs\" (UniqueName: \"kubernetes.io/projected/998df865-bd61-46c1-a57a-de1ecc627326-kube-api-access-ltqzs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vckt2\" (UID: \"998df865-bd61-46c1-a57a-de1ecc627326\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.810588 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.810493    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:10.950344 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:10.950311    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"]
Jul 29 19:15:10.953148 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:15:10.953120    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998df865_bd61_46c1_a57a_de1ecc627326.slice/crio-200aa1b0122e1b110360dbbfe13f7314f31c070942f10d9bf79faae25d70572d WatchSource:0}: Error finding container 200aa1b0122e1b110360dbbfe13f7314f31c070942f10d9bf79faae25d70572d: Status 404 returned error can't find the container with id 200aa1b0122e1b110360dbbfe13f7314f31c070942f10d9bf79faae25d70572d
Jul 29 19:15:11.904626 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:11.904586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2" event={"ID":"998df865-bd61-46c1-a57a-de1ecc627326","Type":"ContainerStarted","Data":"e1ee89a09c43b0070cb11b5ab51affc82585f7405d92a20469f1ffdf529beed6"}
Jul 29 19:15:11.905030 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:11.904684    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2" event={"ID":"998df865-bd61-46c1-a57a-de1ecc627326","Type":"ContainerStarted","Data":"200aa1b0122e1b110360dbbfe13f7314f31c070942f10d9bf79faae25d70572d"}
Jul 29 19:15:11.926760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:11.926711    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2" podStartSLOduration=1.367668669 podStartE2EDuration="1.926695264s" totalImagesPullingTime="559.026595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:15:11.924819755 +0000 UTC m=+878.224492267" watchObservedRunningTime="2026-07-29 19:15:11.926695264 +0000 UTC m=+878.226367765"
Jul 29 19:15:20.810971 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:20.810937    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:20.811392 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:20.810994    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:20.823462 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:20.823431    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:20.959169 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:20.959143    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vckt2"
Jul 29 19:15:34.148476 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:34.148394    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:15:34.152653 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:15:34.152631    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:16:31.009948 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.009912    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7475ccd557-46825"]
Jul 29 19:16:31.010523 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.010129    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7475ccd557-46825" podUID="ba00e9c1-80cf-418a-9467-91c8c18a41aa" containerName="authorino" containerID="cri-o://70b1bc1ad6ed767e9a91491a347dabd8ead29956c9f259a18895293d29804139" gracePeriod=30
Jul 29 19:16:31.260302 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.260230    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7475ccd557-46825"
Jul 29 19:16:31.287573 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.287527    2579 generic.go:356] "Generic (PLEG): container finished" podID="ba00e9c1-80cf-418a-9467-91c8c18a41aa" containerID="70b1bc1ad6ed767e9a91491a347dabd8ead29956c9f259a18895293d29804139" exitCode=0
Jul 29 19:16:31.287745 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.287641    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7475ccd557-46825"
Jul 29 19:16:31.287745 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.287639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7475ccd557-46825" event={"ID":"ba00e9c1-80cf-418a-9467-91c8c18a41aa","Type":"ContainerDied","Data":"70b1bc1ad6ed767e9a91491a347dabd8ead29956c9f259a18895293d29804139"}
Jul 29 19:16:31.287745 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.287695    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7475ccd557-46825" event={"ID":"ba00e9c1-80cf-418a-9467-91c8c18a41aa","Type":"ContainerDied","Data":"72efee6a941255c4913399d1b20ffecdf7ba7112edc4ff17a7ced1e207534f7b"}
Jul 29 19:16:31.287745 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.287717    2579 scope.go:117] "RemoveContainer" containerID="70b1bc1ad6ed767e9a91491a347dabd8ead29956c9f259a18895293d29804139"
Jul 29 19:16:31.297365 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.297338    2579 scope.go:117] "RemoveContainer" containerID="70b1bc1ad6ed767e9a91491a347dabd8ead29956c9f259a18895293d29804139"
Jul 29 19:16:31.297715 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:16:31.297696    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b1bc1ad6ed767e9a91491a347dabd8ead29956c9f259a18895293d29804139\": container with ID starting with 70b1bc1ad6ed767e9a91491a347dabd8ead29956c9f259a18895293d29804139 not found: ID does not exist" containerID="70b1bc1ad6ed767e9a91491a347dabd8ead29956c9f259a18895293d29804139"
Jul 29 19:16:31.297784 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.297731    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b1bc1ad6ed767e9a91491a347dabd8ead29956c9f259a18895293d29804139"} err="failed to get container status \"70b1bc1ad6ed767e9a91491a347dabd8ead29956c9f259a18895293d29804139\": rpc error: code = NotFound desc = could not find container \"70b1bc1ad6ed767e9a91491a347dabd8ead29956c9f259a18895293d29804139\": container with ID starting with 70b1bc1ad6ed767e9a91491a347dabd8ead29956c9f259a18895293d29804139 not found: ID does not exist"
Jul 29 19:16:31.309331 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.309298    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba00e9c1-80cf-418a-9467-91c8c18a41aa-tls-cert\") pod \"ba00e9c1-80cf-418a-9467-91c8c18a41aa\" (UID: \"ba00e9c1-80cf-418a-9467-91c8c18a41aa\") "
Jul 29 19:16:31.309504 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.309346    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxwc8\" (UniqueName: \"kubernetes.io/projected/ba00e9c1-80cf-418a-9467-91c8c18a41aa-kube-api-access-fxwc8\") pod \"ba00e9c1-80cf-418a-9467-91c8c18a41aa\" (UID: \"ba00e9c1-80cf-418a-9467-91c8c18a41aa\") "
Jul 29 19:16:31.311952 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.311915    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba00e9c1-80cf-418a-9467-91c8c18a41aa-kube-api-access-fxwc8" (OuterVolumeSpecName: "kube-api-access-fxwc8") pod "ba00e9c1-80cf-418a-9467-91c8c18a41aa" (UID: "ba00e9c1-80cf-418a-9467-91c8c18a41aa"). InnerVolumeSpecName "kube-api-access-fxwc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:16:31.321848 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.321823    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba00e9c1-80cf-418a-9467-91c8c18a41aa-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ba00e9c1-80cf-418a-9467-91c8c18a41aa" (UID: "ba00e9c1-80cf-418a-9467-91c8c18a41aa"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:16:31.410089 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.410053    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba00e9c1-80cf-418a-9467-91c8c18a41aa-tls-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:16:31.410089 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.410082    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxwc8\" (UniqueName: \"kubernetes.io/projected/ba00e9c1-80cf-418a-9467-91c8c18a41aa-kube-api-access-fxwc8\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:16:31.613326 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.613296    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7475ccd557-46825"]
Jul 29 19:16:31.617249 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:31.617223    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7475ccd557-46825"]
Jul 29 19:16:32.196013 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:16:32.195977    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba00e9c1-80cf-418a-9467-91c8c18a41aa" path="/var/lib/kubelet/pods/ba00e9c1-80cf-418a-9467-91c8c18a41aa/volumes"
Jul 29 19:20:34.178541 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:20:34.178508    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:20:34.185504 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:20:34.185476    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:25:34.207534 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:25:34.207503    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:25:34.216309 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:25:34.216288    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:27:34.311848 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.311770    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"]
Jul 29 19:27:34.312307 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.312169    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba00e9c1-80cf-418a-9467-91c8c18a41aa" containerName="authorino"
Jul 29 19:27:34.312307 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.312181    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba00e9c1-80cf-418a-9467-91c8c18a41aa" containerName="authorino"
Jul 29 19:27:34.312307 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.312269    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba00e9c1-80cf-418a-9467-91c8c18a41aa" containerName="authorino"
Jul 29 19:27:34.314350 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.314334    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.317825 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.317792    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-ca8a6ac3-kserve-self-signed-certs\""
Jul 29 19:27:34.333295 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.333270    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"]
Jul 29 19:27:34.384450 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.384416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-home\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.384625 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.384467    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-model-cache\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.384625 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.384506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-tls-certs\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.384625 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.384541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-dshm\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.384625 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.384583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-tmp-dir\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.384625 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.384606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmlbr\" (UniqueName: \"kubernetes.io/projected/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-kube-api-access-lmlbr\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.485362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.485330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-dshm\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.485532 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.485372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-tmp-dir\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.485532 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.485397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmlbr\" (UniqueName: \"kubernetes.io/projected/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-kube-api-access-lmlbr\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.485532 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.485429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-home\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.485532 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.485454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-model-cache\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.485532 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.485499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-tls-certs\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.485809 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.485789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-tmp-dir\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.485869 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.485844    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-model-cache\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.485910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.485880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-home\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.487689 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.487670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-dshm\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.488067 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.488047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-tls-certs\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.493771 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.493750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmlbr\" (UniqueName: \"kubernetes.io/projected/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-kube-api-access-lmlbr\") pod \"e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") " pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.627308 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.627242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:34.765162 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:34.765136    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"]
Jul 29 19:27:34.766221 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:27:34.766171    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6eb1f39_bbf0_4751_b6ce_87037d300bfa.slice/crio-151bcb6f0b1352e6b18c53fcd0308140178d28fbb1acc8190ae2c2f7d29f822d WatchSource:0}: Error finding container 151bcb6f0b1352e6b18c53fcd0308140178d28fbb1acc8190ae2c2f7d29f822d: Status 404 returned error can't find the container with id 151bcb6f0b1352e6b18c53fcd0308140178d28fbb1acc8190ae2c2f7d29f822d
Jul 29 19:27:35.469815 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:35.469777    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4" event={"ID":"a6eb1f39-bbf0-4751-b6ce-87037d300bfa","Type":"ContainerStarted","Data":"ca030e585132f5250fe6c9c2a1210b3180e32bebc1c04bb3d6eb7ab65e4a4c5a"}
Jul 29 19:27:35.470232 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:35.469825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4" event={"ID":"a6eb1f39-bbf0-4751-b6ce-87037d300bfa","Type":"ContainerStarted","Data":"151bcb6f0b1352e6b18c53fcd0308140178d28fbb1acc8190ae2c2f7d29f822d"}
Jul 29 19:27:35.488828 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:35.488786    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4" podStartSLOduration=1.488773568 podStartE2EDuration="1.488773568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:27:35.488606895 +0000 UTC m=+1621.788279397" watchObservedRunningTime="2026-07-29 19:27:35.488773568 +0000 UTC m=+1621.788446068"
Jul 29 19:27:37.790157 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.790125    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"]
Jul 29 19:27:37.793065 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.793048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.796446 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.796424    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-ca8a6ac3-kserve-self-signed-certs\""
Jul 29 19:27:37.800645 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.800622    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"]
Jul 29 19:27:37.814783 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.814760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-dshm\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.814921 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.814797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-tls-certs\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.814921 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.814849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-model-cache\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.814921 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.814913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-tmp-dir\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.815031 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.814996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njcr8\" (UniqueName: \"kubernetes.io/projected/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-kube-api-access-njcr8\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.815071 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.815047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-home\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.915885 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.915848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njcr8\" (UniqueName: \"kubernetes.io/projected/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-kube-api-access-njcr8\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.916069 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.915896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-home\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.916069 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.915938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-dshm\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.916069 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.915961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-tls-certs\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.916069 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.916012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-model-cache\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.916286 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.916078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-tmp-dir\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.916427 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.916404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-home\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.916517 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.916499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-tmp-dir\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.916561 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.916515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-model-cache\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.918657 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.918636    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-dshm\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.918788 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.918744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-tls-certs\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:37.924439 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:37.924418    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njcr8\" (UniqueName: \"kubernetes.io/projected/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-kube-api-access-njcr8\") pod \"e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") " pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:38.105234 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:38.105113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:38.234500 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:38.234475    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"]
Jul 29 19:27:38.235825 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:27:38.235791    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d124314_9d95_40a8_9cbe_c17bbb4a8d51.slice/crio-07f75d7946a7dea96d6381d6554196aba3cdb52539886b53e1ec2e6d529509df WatchSource:0}: Error finding container 07f75d7946a7dea96d6381d6554196aba3cdb52539886b53e1ec2e6d529509df: Status 404 returned error can't find the container with id 07f75d7946a7dea96d6381d6554196aba3cdb52539886b53e1ec2e6d529509df
Jul 29 19:27:38.485851 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:38.485813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq" event={"ID":"3d124314-9d95-40a8-9cbe-c17bbb4a8d51","Type":"ContainerStarted","Data":"42e41a69a82751ee4220ad937c849e7da335b288aacd495dfb74bd4995c280f2"}
Jul 29 19:27:38.486057 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:38.485864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq" event={"ID":"3d124314-9d95-40a8-9cbe-c17bbb4a8d51","Type":"ContainerStarted","Data":"07f75d7946a7dea96d6381d6554196aba3cdb52539886b53e1ec2e6d529509df"}
Jul 29 19:27:38.506837 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:38.506787    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq" podStartSLOduration=1.5067742100000001 podStartE2EDuration="1.50677421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:27:38.505115889 +0000 UTC m=+1624.804788389" watchObservedRunningTime="2026-07-29 19:27:38.50677421 +0000 UTC m=+1624.806446709"
Jul 29 19:27:44.628322 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:44.628225    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:44.628322 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:44.628288    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:44.640596 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:44.640574    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:45.529856 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:45.529828    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:27:48.105699 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:48.105663    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:48.106121 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:48.105717    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:48.118038 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:48.118016    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:27:48.544768 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:27:48.544742    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:28:08.589867 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:08.589835    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"]
Jul 29 19:28:09.626017 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:09.625983    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"]
Jul 29 19:28:23.609069 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.609029    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4" podUID="a6eb1f39-bbf0-4751-b6ce-87037d300bfa" containerName="main" containerID="cri-o://ca030e585132f5250fe6c9c2a1210b3180e32bebc1c04bb3d6eb7ab65e4a4c5a" gracePeriod=45
Jul 29 19:28:23.857655 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.857632    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:28:23.921802 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.921779    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-home\") pod \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") "
Jul 29 19:28:23.922119 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.921833    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-tmp-dir\") pod \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") "
Jul 29 19:28:23.922119 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.921854    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-model-cache\") pod \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") "
Jul 29 19:28:23.922119 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.921897    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmlbr\" (UniqueName: \"kubernetes.io/projected/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-kube-api-access-lmlbr\") pod \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") "
Jul 29 19:28:23.922119 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.921923    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-tls-certs\") pod \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") "
Jul 29 19:28:23.922119 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.921958    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-dshm\") pod \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\" (UID: \"a6eb1f39-bbf0-4751-b6ce-87037d300bfa\") "
Jul 29 19:28:23.922119 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.922027    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-home" (OuterVolumeSpecName: "home") pod "a6eb1f39-bbf0-4751-b6ce-87037d300bfa" (UID: "a6eb1f39-bbf0-4751-b6ce-87037d300bfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:28:23.922423 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.922124    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6eb1f39-bbf0-4751-b6ce-87037d300bfa" (UID: "a6eb1f39-bbf0-4751-b6ce-87037d300bfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:28:23.922423 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.922139    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6eb1f39-bbf0-4751-b6ce-87037d300bfa" (UID: "a6eb1f39-bbf0-4751-b6ce-87037d300bfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:28:23.922423 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.922286    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-home\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:28:23.922423 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.922299    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-tmp-dir\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:28:23.922423 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.922308    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-model-cache\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:28:23.924184 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.924155    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6eb1f39-bbf0-4751-b6ce-87037d300bfa" (UID: "a6eb1f39-bbf0-4751-b6ce-87037d300bfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:28:23.924291 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.924241    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-kube-api-access-lmlbr" (OuterVolumeSpecName: "kube-api-access-lmlbr") pod "a6eb1f39-bbf0-4751-b6ce-87037d300bfa" (UID: "a6eb1f39-bbf0-4751-b6ce-87037d300bfa"). InnerVolumeSpecName "kube-api-access-lmlbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:28:23.924291 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:23.924277    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-dshm" (OuterVolumeSpecName: "dshm") pod "a6eb1f39-bbf0-4751-b6ce-87037d300bfa" (UID: "a6eb1f39-bbf0-4751-b6ce-87037d300bfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:28:24.022781 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.022758    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-tls-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:28:24.022781 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.022778    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-dshm\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:28:24.022902 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.022787    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmlbr\" (UniqueName: \"kubernetes.io/projected/a6eb1f39-bbf0-4751-b6ce-87037d300bfa-kube-api-access-lmlbr\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:28:24.645701 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.645644    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq" podUID="3d124314-9d95-40a8-9cbe-c17bbb4a8d51" containerName="main" containerID="cri-o://42e41a69a82751ee4220ad937c849e7da335b288aacd495dfb74bd4995c280f2" gracePeriod=45
Jul 29 19:28:24.714554 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.714527    2579 generic.go:356] "Generic (PLEG): container finished" podID="a6eb1f39-bbf0-4751-b6ce-87037d300bfa" containerID="ca030e585132f5250fe6c9c2a1210b3180e32bebc1c04bb3d6eb7ab65e4a4c5a" exitCode=0
Jul 29 19:28:24.714643 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.714575    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4" event={"ID":"a6eb1f39-bbf0-4751-b6ce-87037d300bfa","Type":"ContainerDied","Data":"ca030e585132f5250fe6c9c2a1210b3180e32bebc1c04bb3d6eb7ab65e4a4c5a"}
Jul 29 19:28:24.714643 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.714582    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"
Jul 29 19:28:24.714643 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.714601    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4" event={"ID":"a6eb1f39-bbf0-4751-b6ce-87037d300bfa","Type":"ContainerDied","Data":"151bcb6f0b1352e6b18c53fcd0308140178d28fbb1acc8190ae2c2f7d29f822d"}
Jul 29 19:28:24.714643 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.714617    2579 scope.go:117] "RemoveContainer" containerID="ca030e585132f5250fe6c9c2a1210b3180e32bebc1c04bb3d6eb7ab65e4a4c5a"
Jul 29 19:28:24.732089 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.732067    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"]
Jul 29 19:28:24.735335 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.735315    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-ca8a6ac3-kserve-775bdd6887-fblq4"]
Jul 29 19:28:24.756080 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.756062    2579 scope.go:117] "RemoveContainer" containerID="ca030e585132f5250fe6c9c2a1210b3180e32bebc1c04bb3d6eb7ab65e4a4c5a"
Jul 29 19:28:24.756383 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:28:24.756365    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca030e585132f5250fe6c9c2a1210b3180e32bebc1c04bb3d6eb7ab65e4a4c5a\": container with ID starting with ca030e585132f5250fe6c9c2a1210b3180e32bebc1c04bb3d6eb7ab65e4a4c5a not found: ID does not exist" containerID="ca030e585132f5250fe6c9c2a1210b3180e32bebc1c04bb3d6eb7ab65e4a4c5a"
Jul 29 19:28:24.756454 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.756391    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca030e585132f5250fe6c9c2a1210b3180e32bebc1c04bb3d6eb7ab65e4a4c5a"} err="failed to get container status \"ca030e585132f5250fe6c9c2a1210b3180e32bebc1c04bb3d6eb7ab65e4a4c5a\": rpc error: code = NotFound desc = could not find container \"ca030e585132f5250fe6c9c2a1210b3180e32bebc1c04bb3d6eb7ab65e4a4c5a\": container with ID starting with ca030e585132f5250fe6c9c2a1210b3180e32bebc1c04bb3d6eb7ab65e4a4c5a not found: ID does not exist"
Jul 29 19:28:24.885968 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.885943    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:28:24.930587 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.930537    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-home\") pod \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") "
Jul 29 19:28:24.930688 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.930606    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njcr8\" (UniqueName: \"kubernetes.io/projected/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-kube-api-access-njcr8\") pod \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") "
Jul 29 19:28:24.930688 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.930655    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-tls-certs\") pod \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") "
Jul 29 19:28:24.930688 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.930682    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-model-cache\") pod \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") "
Jul 29 19:28:24.930779 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.930719    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-dshm\") pod \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") "
Jul 29 19:28:24.930816 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.930790    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-tmp-dir\") pod \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\" (UID: \"3d124314-9d95-40a8-9cbe-c17bbb4a8d51\") "
Jul 29 19:28:24.930816 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.930798    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-home" (OuterVolumeSpecName: "home") pod "3d124314-9d95-40a8-9cbe-c17bbb4a8d51" (UID: "3d124314-9d95-40a8-9cbe-c17bbb4a8d51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:28:24.931015 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.930990    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d124314-9d95-40a8-9cbe-c17bbb4a8d51" (UID: "3d124314-9d95-40a8-9cbe-c17bbb4a8d51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:28:24.931015 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.931005    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d124314-9d95-40a8-9cbe-c17bbb4a8d51" (UID: "3d124314-9d95-40a8-9cbe-c17bbb4a8d51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:28:24.931176 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.931148    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-tmp-dir\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:28:24.931176 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.931173    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-home\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:28:24.931176 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.931182    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-model-cache\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:28:24.932889 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.932868    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-dshm" (OuterVolumeSpecName: "dshm") pod "3d124314-9d95-40a8-9cbe-c17bbb4a8d51" (UID: "3d124314-9d95-40a8-9cbe-c17bbb4a8d51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:28:24.932889 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.932889    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-kube-api-access-njcr8" (OuterVolumeSpecName: "kube-api-access-njcr8") pod "3d124314-9d95-40a8-9cbe-c17bbb4a8d51" (UID: "3d124314-9d95-40a8-9cbe-c17bbb4a8d51"). InnerVolumeSpecName "kube-api-access-njcr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:28:24.933014 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:24.932903    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d124314-9d95-40a8-9cbe-c17bbb4a8d51" (UID: "3d124314-9d95-40a8-9cbe-c17bbb4a8d51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:28:25.032538 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:25.032522    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njcr8\" (UniqueName: \"kubernetes.io/projected/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-kube-api-access-njcr8\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:28:25.032538 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:25.032540    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-tls-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:28:25.032662 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:25.032548    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d124314-9d95-40a8-9cbe-c17bbb4a8d51-dshm\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:28:25.720647 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:25.720618    2579 generic.go:356] "Generic (PLEG): container finished" podID="3d124314-9d95-40a8-9cbe-c17bbb4a8d51" containerID="42e41a69a82751ee4220ad937c849e7da335b288aacd495dfb74bd4995c280f2" exitCode=0
Jul 29 19:28:25.721054 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:25.720672    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"
Jul 29 19:28:25.721054 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:25.720701    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq" event={"ID":"3d124314-9d95-40a8-9cbe-c17bbb4a8d51","Type":"ContainerDied","Data":"42e41a69a82751ee4220ad937c849e7da335b288aacd495dfb74bd4995c280f2"}
Jul 29 19:28:25.721054 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:25.720751    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq" event={"ID":"3d124314-9d95-40a8-9cbe-c17bbb4a8d51","Type":"ContainerDied","Data":"07f75d7946a7dea96d6381d6554196aba3cdb52539886b53e1ec2e6d529509df"}
Jul 29 19:28:25.721054 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:25.720778    2579 scope.go:117] "RemoveContainer" containerID="42e41a69a82751ee4220ad937c849e7da335b288aacd495dfb74bd4995c280f2"
Jul 29 19:28:25.729283 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:25.729266    2579 scope.go:117] "RemoveContainer" containerID="42e41a69a82751ee4220ad937c849e7da335b288aacd495dfb74bd4995c280f2"
Jul 29 19:28:25.729557 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:28:25.729536    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e41a69a82751ee4220ad937c849e7da335b288aacd495dfb74bd4995c280f2\": container with ID starting with 42e41a69a82751ee4220ad937c849e7da335b288aacd495dfb74bd4995c280f2 not found: ID does not exist" containerID="42e41a69a82751ee4220ad937c849e7da335b288aacd495dfb74bd4995c280f2"
Jul 29 19:28:25.729611 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:25.729572    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e41a69a82751ee4220ad937c849e7da335b288aacd495dfb74bd4995c280f2"} err="failed to get container status \"42e41a69a82751ee4220ad937c849e7da335b288aacd495dfb74bd4995c280f2\": rpc error: code = NotFound desc = could not find container \"42e41a69a82751ee4220ad937c849e7da335b288aacd495dfb74bd4995c280f2\": container with ID starting with 42e41a69a82751ee4220ad937c849e7da335b288aacd495dfb74bd4995c280f2 not found: ID does not exist"
Jul 29 19:28:25.745265 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:25.745246    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"]
Jul 29 19:28:25.751370 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:25.751352    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-ca8a6ac3-kserve-55bb7454b7-r8nrq"]
Jul 29 19:28:26.197008 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:26.196987    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d124314-9d95-40a8-9cbe-c17bbb4a8d51" path="/var/lib/kubelet/pods/3d124314-9d95-40a8-9cbe-c17bbb4a8d51/volumes"
Jul 29 19:28:26.197352 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:28:26.197340    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6eb1f39-bbf0-4751-b6ce-87037d300bfa" path="/var/lib/kubelet/pods/a6eb1f39-bbf0-4751-b6ce-87037d300bfa/volumes"
Jul 29 19:30:00.150854 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:00.149518    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6wddw"]
Jul 29 19:30:00.150854 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:00.150399    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d124314-9d95-40a8-9cbe-c17bbb4a8d51" containerName="main"
Jul 29 19:30:00.150854 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:00.150418    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d124314-9d95-40a8-9cbe-c17bbb4a8d51" containerName="main"
Jul 29 19:30:00.150854 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:00.150444    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6eb1f39-bbf0-4751-b6ce-87037d300bfa" containerName="main"
Jul 29 19:30:00.150854 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:00.150459    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6eb1f39-bbf0-4751-b6ce-87037d300bfa" containerName="main"
Jul 29 19:30:00.150854 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:00.150640    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d124314-9d95-40a8-9cbe-c17bbb4a8d51" containerName="main"
Jul 29 19:30:00.150854 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:00.150653    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6eb1f39-bbf0-4751-b6ce-87037d300bfa" containerName="main"
Jul 29 19:30:00.155468 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:00.155445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6wddw"
Jul 29 19:30:00.158885 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:00.158865    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq9q2\""
Jul 29 19:30:00.164771 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:00.164756    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6wddw"
Jul 29 19:30:00.174133 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:00.174109    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6wddw"]
Jul 29 19:30:00.303078 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:00.303044    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6wddw"]
Jul 29 19:30:00.303584 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:30:00.303554    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181efb04_abbb_4abd_9dfd_5bfe475b9565.slice/crio-5270277cfb198129eb9b90c89944fd4ce563f5b14c65ba7bcf05529b155a1846 WatchSource:0}: Error finding container 5270277cfb198129eb9b90c89944fd4ce563f5b14c65ba7bcf05529b155a1846: Status 404 returned error can't find the container with id 5270277cfb198129eb9b90c89944fd4ce563f5b14c65ba7bcf05529b155a1846
Jul 29 19:30:01.160036 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:01.160003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6wddw" event={"ID":"181efb04-abbb-4abd-9dfd-5bfe475b9565","Type":"ContainerStarted","Data":"7489c90020a1ef4a29f32dd5152b966cccc7f7f1aa489680fc85ed80ce8a1a71"}
Jul 29 19:30:01.160453 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:01.160055    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6wddw" event={"ID":"181efb04-abbb-4abd-9dfd-5bfe475b9565","Type":"ContainerStarted","Data":"5270277cfb198129eb9b90c89944fd4ce563f5b14c65ba7bcf05529b155a1846"}
Jul 29 19:30:01.176300 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:01.176250    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6wddw" podStartSLOduration=1.176234863 podStartE2EDuration="1.176234863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:01.175647851 +0000 UTC m=+1767.475320351" watchObservedRunningTime="2026-07-29 19:30:01.176234863 +0000 UTC m=+1767.475907363"
Jul 29 19:30:02.166421 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:02.166383    2579 generic.go:356] "Generic (PLEG): container finished" podID="181efb04-abbb-4abd-9dfd-5bfe475b9565" containerID="7489c90020a1ef4a29f32dd5152b966cccc7f7f1aa489680fc85ed80ce8a1a71" exitCode=0
Jul 29 19:30:02.166799 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:02.166420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6wddw" event={"ID":"181efb04-abbb-4abd-9dfd-5bfe475b9565","Type":"ContainerDied","Data":"7489c90020a1ef4a29f32dd5152b966cccc7f7f1aa489680fc85ed80ce8a1a71"}
Jul 29 19:30:03.301290 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:03.301264    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6wddw"
Jul 29 19:30:04.176517 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:04.176491    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6wddw"
Jul 29 19:30:04.176660 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:04.176524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6wddw" event={"ID":"181efb04-abbb-4abd-9dfd-5bfe475b9565","Type":"ContainerDied","Data":"5270277cfb198129eb9b90c89944fd4ce563f5b14c65ba7bcf05529b155a1846"}
Jul 29 19:30:04.176660 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:04.176570    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5270277cfb198129eb9b90c89944fd4ce563f5b14c65ba7bcf05529b155a1846"
Jul 29 19:30:34.235846 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:34.235817    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:30:34.247299 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:30:34.247275    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:31:52.908665 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:52.908619    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"]
Jul 29 19:31:52.909324 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:52.909230    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="181efb04-abbb-4abd-9dfd-5bfe475b9565" containerName="cleanup"
Jul 29 19:31:52.909324 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:52.909252    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="181efb04-abbb-4abd-9dfd-5bfe475b9565" containerName="cleanup"
Jul 29 19:31:52.909454 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:52.909344    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="181efb04-abbb-4abd-9dfd-5bfe475b9565" containerName="cleanup"
Jul 29 19:31:52.912509 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:52.912490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:31:52.915370 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:52.915348    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:31:52.915472 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:52.915383    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5bzmf\""
Jul 29 19:31:52.916671 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:52.916651    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:31:52.916800 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:52.916655    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a448180e-serving-cert\""
Jul 29 19:31:52.924799 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:52.924777    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"]
Jul 29 19:31:52.930564 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:52.930543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517c315b-7068-455b-9b49-2eec73e7a6ce-maas-api-tls\") pod \"maas-api-e2e-ait-a448180e-78d56765cb-cbtxb\" (UID: \"517c315b-7068-455b-9b49-2eec73e7a6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:31:52.930667 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:52.930603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517c315b-7068-455b-9b49-2eec73e7a6ce-openshift-service-ca\") pod \"maas-api-e2e-ait-a448180e-78d56765cb-cbtxb\" (UID: \"517c315b-7068-455b-9b49-2eec73e7a6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:31:52.930667 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:52.930630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmr82\" (UniqueName: \"kubernetes.io/projected/517c315b-7068-455b-9b49-2eec73e7a6ce-kube-api-access-jmr82\") pod \"maas-api-e2e-ait-a448180e-78d56765cb-cbtxb\" (UID: \"517c315b-7068-455b-9b49-2eec73e7a6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:31:53.031634 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.031595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517c315b-7068-455b-9b49-2eec73e7a6ce-maas-api-tls\") pod \"maas-api-e2e-ait-a448180e-78d56765cb-cbtxb\" (UID: \"517c315b-7068-455b-9b49-2eec73e7a6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:31:53.031836 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.031675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517c315b-7068-455b-9b49-2eec73e7a6ce-openshift-service-ca\") pod \"maas-api-e2e-ait-a448180e-78d56765cb-cbtxb\" (UID: \"517c315b-7068-455b-9b49-2eec73e7a6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:31:53.031836 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.031715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmr82\" (UniqueName: \"kubernetes.io/projected/517c315b-7068-455b-9b49-2eec73e7a6ce-kube-api-access-jmr82\") pod \"maas-api-e2e-ait-a448180e-78d56765cb-cbtxb\" (UID: \"517c315b-7068-455b-9b49-2eec73e7a6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:31:53.031836 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:31:53.031764    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-serving-cert: secret "maas-api-e2e-ait-a448180e-serving-cert" not found
Jul 29 19:31:53.031956 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:31:53.031842    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/517c315b-7068-455b-9b49-2eec73e7a6ce-maas-api-tls podName:517c315b-7068-455b-9b49-2eec73e7a6ce nodeName:}" failed. No retries permitted until 2026-07-29 19:31:53.531819957 +0000 UTC m=+1879.831492441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/517c315b-7068-455b-9b49-2eec73e7a6ce-maas-api-tls") pod "maas-api-e2e-ait-a448180e-78d56765cb-cbtxb" (UID: "517c315b-7068-455b-9b49-2eec73e7a6ce") : secret "maas-api-e2e-ait-a448180e-serving-cert" not found
Jul 29 19:31:53.032386 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.032365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517c315b-7068-455b-9b49-2eec73e7a6ce-openshift-service-ca\") pod \"maas-api-e2e-ait-a448180e-78d56765cb-cbtxb\" (UID: \"517c315b-7068-455b-9b49-2eec73e7a6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:31:53.041540 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.041512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmr82\" (UniqueName: \"kubernetes.io/projected/517c315b-7068-455b-9b49-2eec73e7a6ce-kube-api-access-jmr82\") pod \"maas-api-e2e-ait-a448180e-78d56765cb-cbtxb\" (UID: \"517c315b-7068-455b-9b49-2eec73e7a6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:31:53.532626 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.532592    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"]
Jul 29 19:31:53.536294 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.536267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517c315b-7068-455b-9b49-2eec73e7a6ce-maas-api-tls\") pod \"maas-api-e2e-ait-a448180e-78d56765cb-cbtxb\" (UID: \"517c315b-7068-455b-9b49-2eec73e7a6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:31:53.536441 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.536387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"
Jul 29 19:31:53.538836 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.538816    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517c315b-7068-455b-9b49-2eec73e7a6ce-maas-api-tls\") pod \"maas-api-e2e-ait-a448180e-78d56765cb-cbtxb\" (UID: \"517c315b-7068-455b-9b49-2eec73e7a6ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:31:53.539170 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.539152    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a448180e\""
Jul 29 19:31:53.549238 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.549215    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"]
Jul 29 19:31:53.552940 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.552919    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"]
Jul 29 19:31:53.553043 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.552996    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"
Jul 29 19:31:53.556083 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.556064    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a448180e-dockercfg-226vk\""
Jul 29 19:31:53.561378 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.561357    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"]
Jul 29 19:31:53.637697 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.637668    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjdhs\" (UniqueName: \"kubernetes.io/projected/f9db3ebf-202e-4db5-93e7-f7c94407be15-kube-api-access-qjdhs\") pod \"payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb\" (UID: \"f9db3ebf-202e-4db5-93e7-f7c94407be15\") " pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"
Jul 29 19:31:53.637826 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.637707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf8w6\" (UniqueName: \"kubernetes.io/projected/e82214cf-4132-41b4-8807-84773974e8a6-kube-api-access-pf8w6\") pod \"payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd\" (UID: \"e82214cf-4132-41b4-8807-84773974e8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"
Jul 29 19:31:53.637826 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.637735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82214cf-4132-41b4-8807-84773974e8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd\" (UID: \"e82214cf-4132-41b4-8807-84773974e8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"
Jul 29 19:31:53.637896 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.637834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9db3ebf-202e-4db5-93e7-f7c94407be15-plugins-config-volume\") pod \"payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb\" (UID: \"f9db3ebf-202e-4db5-93e7-f7c94407be15\") " pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"
Jul 29 19:31:53.739176 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.739149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf8w6\" (UniqueName: \"kubernetes.io/projected/e82214cf-4132-41b4-8807-84773974e8a6-kube-api-access-pf8w6\") pod \"payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd\" (UID: \"e82214cf-4132-41b4-8807-84773974e8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"
Jul 29 19:31:53.739357 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.739224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82214cf-4132-41b4-8807-84773974e8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd\" (UID: \"e82214cf-4132-41b4-8807-84773974e8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"
Jul 29 19:31:53.739357 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.739291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9db3ebf-202e-4db5-93e7-f7c94407be15-plugins-config-volume\") pod \"payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb\" (UID: \"f9db3ebf-202e-4db5-93e7-f7c94407be15\") " pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"
Jul 29 19:31:53.739357 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.739338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjdhs\" (UniqueName: \"kubernetes.io/projected/f9db3ebf-202e-4db5-93e7-f7c94407be15-kube-api-access-qjdhs\") pod \"payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb\" (UID: \"f9db3ebf-202e-4db5-93e7-f7c94407be15\") " pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"
Jul 29 19:31:53.739955 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.739930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9db3ebf-202e-4db5-93e7-f7c94407be15-plugins-config-volume\") pod \"payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb\" (UID: \"f9db3ebf-202e-4db5-93e7-f7c94407be15\") " pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"
Jul 29 19:31:53.740043 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.739939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82214cf-4132-41b4-8807-84773974e8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd\" (UID: \"e82214cf-4132-41b4-8807-84773974e8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"
Jul 29 19:31:53.748355 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.748335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf8w6\" (UniqueName: \"kubernetes.io/projected/e82214cf-4132-41b4-8807-84773974e8a6-kube-api-access-pf8w6\") pod \"payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd\" (UID: \"e82214cf-4132-41b4-8807-84773974e8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"
Jul 29 19:31:53.749476 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.749457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjdhs\" (UniqueName: \"kubernetes.io/projected/f9db3ebf-202e-4db5-93e7-f7c94407be15-kube-api-access-qjdhs\") pod \"payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb\" (UID: \"f9db3ebf-202e-4db5-93e7-f7c94407be15\") " pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"
Jul 29 19:31:53.823823 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.823749    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:31:53.856119 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.856090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"
Jul 29 19:31:53.866298 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.866270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"
Jul 29 19:31:53.978272 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.978243    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"]
Jul 29 19:31:53.980428 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:31:53.980402    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517c315b_7068_455b_9b49_2eec73e7a6ce.slice/crio-47c885bc4c6ba20bc6d0f1f7444b6426bfd5b3059aa7c9bc9336dbf36a886e1e WatchSource:0}: Error finding container 47c885bc4c6ba20bc6d0f1f7444b6426bfd5b3059aa7c9bc9336dbf36a886e1e: Status 404 returned error can't find the container with id 47c885bc4c6ba20bc6d0f1f7444b6426bfd5b3059aa7c9bc9336dbf36a886e1e
Jul 29 19:31:53.981785 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:53.981769    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:31:54.234964 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:54.234936    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"]
Jul 29 19:31:54.237207 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:31:54.237154    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9db3ebf_202e_4db5_93e7_f7c94407be15.slice/crio-85296d953983c6c178a1ba3ac6aa7083ca70ef630d336fc675a51add2c0d376d WatchSource:0}: Error finding container 85296d953983c6c178a1ba3ac6aa7083ca70ef630d336fc675a51add2c0d376d: Status 404 returned error can't find the container with id 85296d953983c6c178a1ba3ac6aa7083ca70ef630d336fc675a51add2c0d376d
Jul 29 19:31:54.265026 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:54.265005    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"]
Jul 29 19:31:54.276296 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:31:54.276268    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82214cf_4132_41b4_8807_84773974e8a6.slice/crio-1ef47a68d232de5fe59d76dbe1497cd3ad5cf2f37a2be0d4df9b70fb3906b6c7 WatchSource:0}: Error finding container 1ef47a68d232de5fe59d76dbe1497cd3ad5cf2f37a2be0d4df9b70fb3906b6c7: Status 404 returned error can't find the container with id 1ef47a68d232de5fe59d76dbe1497cd3ad5cf2f37a2be0d4df9b70fb3906b6c7
Jul 29 19:31:54.709691 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:54.708847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb" event={"ID":"f9db3ebf-202e-4db5-93e7-f7c94407be15","Type":"ContainerStarted","Data":"2c4f464546a2ecb709dc9bd8a9782c1ef16c29761dba7e5f17e861f27bffc7e3"}
Jul 29 19:31:54.709891 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:54.709770    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"
Jul 29 19:31:54.709891 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:54.709804    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb" event={"ID":"f9db3ebf-202e-4db5-93e7-f7c94407be15","Type":"ContainerStarted","Data":"85296d953983c6c178a1ba3ac6aa7083ca70ef630d336fc675a51add2c0d376d"}
Jul 29 19:31:54.711306 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:54.711242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb" event={"ID":"517c315b-7068-455b-9b49-2eec73e7a6ce","Type":"ContainerStarted","Data":"47c885bc4c6ba20bc6d0f1f7444b6426bfd5b3059aa7c9bc9336dbf36a886e1e"}
Jul 29 19:31:54.713223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:54.713020    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd" event={"ID":"e82214cf-4132-41b4-8807-84773974e8a6","Type":"ContainerStarted","Data":"16dc200023054ee538bbfd167cf91b92ebec8e972b77a173a1b32cca5d908731"}
Jul 29 19:31:54.713223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:54.713063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd" event={"ID":"e82214cf-4132-41b4-8807-84773974e8a6","Type":"ContainerStarted","Data":"1ef47a68d232de5fe59d76dbe1497cd3ad5cf2f37a2be0d4df9b70fb3906b6c7"}
Jul 29 19:31:54.713685 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:54.713395    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"
Jul 29 19:31:54.727720 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:54.727662    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb" podStartSLOduration=1.727642366 podStartE2EDuration="1.727642366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:54.726084179 +0000 UTC m=+1881.025756680" watchObservedRunningTime="2026-07-29 19:31:54.727642366 +0000 UTC m=+1881.027314868"
Jul 29 19:31:54.744623 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:54.744574    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd" podStartSLOduration=1.744558433 podStartE2EDuration="1.744558433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:54.742101932 +0000 UTC m=+1881.041774458" watchObservedRunningTime="2026-07-29 19:31:54.744558433 +0000 UTC m=+1881.044230931"
Jul 29 19:31:56.725774 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:56.725693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb" event={"ID":"517c315b-7068-455b-9b49-2eec73e7a6ce","Type":"ContainerStarted","Data":"2b21e808ff03911dc35c460e01f654f93ad6bd12a4b6ce136fac8bd8f0c67514"}
Jul 29 19:31:56.726166 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:56.725924    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:31:56.744915 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:31:56.744870    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb" podStartSLOduration=2.273707463 podStartE2EDuration="4.744856791s" totalImagesPullingTime="2.471149328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:56.742763011 +0000 UTC m=+1883.042435512" watchObservedRunningTime="2026-07-29 19:31:56.744856791 +0000 UTC m=+1883.044529291"
Jul 29 19:32:02.735984 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:02.735955    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:32:05.721380 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:05.721352    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"
Jul 29 19:32:06.728259 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:06.728232    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"
Jul 29 19:32:18.198366 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:18.198335    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp"]
Jul 29 19:32:18.201955 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:18.201940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp"
Jul 29 19:32:18.204821 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:18.204804    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq9q2\""
Jul 29 19:32:18.210151 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:18.210125    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp"]
Jul 29 19:32:18.366600 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:18.361089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72188e14-b439-4962-a359-66c339e047ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a448180e-826bp\" (UID: \"72188e14-b439-4962-a359-66c339e047ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp"
Jul 29 19:32:18.462626 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:18.462529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72188e14-b439-4962-a359-66c339e047ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a448180e-826bp\" (UID: \"72188e14-b439-4962-a359-66c339e047ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp"
Jul 29 19:32:18.463226 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:18.463168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72188e14-b439-4962-a359-66c339e047ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a448180e-826bp\" (UID: \"72188e14-b439-4962-a359-66c339e047ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp"
Jul 29 19:32:18.513977 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:18.513939    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp"
Jul 29 19:32:18.649329 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:18.649304    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp"]
Jul 29 19:32:18.651012 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:32:18.650976    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72188e14_b439_4962_a359_66c339e047ef.slice/crio-228e2607d49b42f3ba76cb8ac0edc463b72adaf0c501594f14a5252da5e0ae9e WatchSource:0}: Error finding container 228e2607d49b42f3ba76cb8ac0edc463b72adaf0c501594f14a5252da5e0ae9e: Status 404 returned error can't find the container with id 228e2607d49b42f3ba76cb8ac0edc463b72adaf0c501594f14a5252da5e0ae9e
Jul 29 19:32:18.831451 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:18.831362    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp" event={"ID":"72188e14-b439-4962-a359-66c339e047ef","Type":"ContainerStarted","Data":"228e2607d49b42f3ba76cb8ac0edc463b72adaf0c501594f14a5252da5e0ae9e"}
Jul 29 19:32:19.837924 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:19.837834    2579 generic.go:356] "Generic (PLEG): container finished" podID="72188e14-b439-4962-a359-66c339e047ef" containerID="8fb34a912a7ef3edd268301e29a0ce539e96d70189c0ad02ebd0ac540e10a065" exitCode=0
Jul 29 19:32:19.838323 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:19.837928    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp" event={"ID":"72188e14-b439-4962-a359-66c339e047ef","Type":"ContainerDied","Data":"8fb34a912a7ef3edd268301e29a0ce539e96d70189c0ad02ebd0ac540e10a065"}
Jul 29 19:32:20.972648 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:20.972624    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp"
Jul 29 19:32:21.084975 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:21.084938    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72188e14-b439-4962-a359-66c339e047ef-maas-api-service-ca\") pod \"72188e14-b439-4962-a359-66c339e047ef\" (UID: \"72188e14-b439-4962-a359-66c339e047ef\") "
Jul 29 19:32:21.085344 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:21.085322    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72188e14-b439-4962-a359-66c339e047ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72188e14-b439-4962-a359-66c339e047ef" (UID: "72188e14-b439-4962-a359-66c339e047ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:21.186639 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:21.186599    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72188e14-b439-4962-a359-66c339e047ef-maas-api-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:21.849113 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:21.849090    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp"
Jul 29 19:32:21.849296 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:21.849087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp" event={"ID":"72188e14-b439-4962-a359-66c339e047ef","Type":"ContainerDied","Data":"228e2607d49b42f3ba76cb8ac0edc463b72adaf0c501594f14a5252da5e0ae9e"}
Jul 29 19:32:21.849296 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:21.849226    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="228e2607d49b42f3ba76cb8ac0edc463b72adaf0c501594f14a5252da5e0ae9e"
Jul 29 19:32:23.329341 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.329310    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"]
Jul 29 19:32:23.329852 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.329512    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb" podUID="517c315b-7068-455b-9b49-2eec73e7a6ce" containerName="maas-api" containerID="cri-o://2b21e808ff03911dc35c460e01f654f93ad6bd12a4b6ce136fac8bd8f0c67514" gracePeriod=30
Jul 29 19:32:23.385223 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.385172    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"]
Jul 29 19:32:23.385433 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.385406    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb" podUID="f9db3ebf-202e-4db5-93e7-f7c94407be15" containerName="payload-processing" containerID="cri-o://2c4f464546a2ecb709dc9bd8a9782c1ef16c29761dba7e5f17e861f27bffc7e3" gracePeriod=30
Jul 29 19:32:23.393355 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.393326    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"]
Jul 29 19:32:23.393570 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.393523    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd" podUID="e82214cf-4132-41b4-8807-84773974e8a6" containerName="payload-pre-processing" containerID="cri-o://16dc200023054ee538bbfd167cf91b92ebec8e972b77a173a1b32cca5d908731" gracePeriod=30
Jul 29 19:32:23.656570 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.656545    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"
Jul 29 19:32:23.664741 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.664711    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:32:23.674567 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.674545    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"
Jul 29 19:32:23.700489 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.700458    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp"]
Jul 29 19:32:23.717022 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.716991    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a448180e-826bp"]
Jul 29 19:32:23.812309 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.812271    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517c315b-7068-455b-9b49-2eec73e7a6ce-maas-api-tls\") pod \"517c315b-7068-455b-9b49-2eec73e7a6ce\" (UID: \"517c315b-7068-455b-9b49-2eec73e7a6ce\") "
Jul 29 19:32:23.812309 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.812310    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmr82\" (UniqueName: \"kubernetes.io/projected/517c315b-7068-455b-9b49-2eec73e7a6ce-kube-api-access-jmr82\") pod \"517c315b-7068-455b-9b49-2eec73e7a6ce\" (UID: \"517c315b-7068-455b-9b49-2eec73e7a6ce\") "
Jul 29 19:32:23.812518 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.812328    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjdhs\" (UniqueName: \"kubernetes.io/projected/f9db3ebf-202e-4db5-93e7-f7c94407be15-kube-api-access-qjdhs\") pod \"f9db3ebf-202e-4db5-93e7-f7c94407be15\" (UID: \"f9db3ebf-202e-4db5-93e7-f7c94407be15\") "
Jul 29 19:32:23.812518 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.812343    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf8w6\" (UniqueName: \"kubernetes.io/projected/e82214cf-4132-41b4-8807-84773974e8a6-kube-api-access-pf8w6\") pod \"e82214cf-4132-41b4-8807-84773974e8a6\" (UID: \"e82214cf-4132-41b4-8807-84773974e8a6\") "
Jul 29 19:32:23.812518 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.812403    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9db3ebf-202e-4db5-93e7-f7c94407be15-plugins-config-volume\") pod \"f9db3ebf-202e-4db5-93e7-f7c94407be15\" (UID: \"f9db3ebf-202e-4db5-93e7-f7c94407be15\") "
Jul 29 19:32:23.812518 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.812451    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82214cf-4132-41b4-8807-84773974e8a6-plugins-config-volume\") pod \"e82214cf-4132-41b4-8807-84773974e8a6\" (UID: \"e82214cf-4132-41b4-8807-84773974e8a6\") "
Jul 29 19:32:23.812518 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.812503    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517c315b-7068-455b-9b49-2eec73e7a6ce-openshift-service-ca\") pod \"517c315b-7068-455b-9b49-2eec73e7a6ce\" (UID: \"517c315b-7068-455b-9b49-2eec73e7a6ce\") "
Jul 29 19:32:23.812869 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.812842    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9db3ebf-202e-4db5-93e7-f7c94407be15-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9db3ebf-202e-4db5-93e7-f7c94407be15" (UID: "f9db3ebf-202e-4db5-93e7-f7c94407be15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:23.813043 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.813010    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82214cf-4132-41b4-8807-84773974e8a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e82214cf-4132-41b4-8807-84773974e8a6" (UID: "e82214cf-4132-41b4-8807-84773974e8a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:23.813131 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.813022    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/517c315b-7068-455b-9b49-2eec73e7a6ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "517c315b-7068-455b-9b49-2eec73e7a6ce" (UID: "517c315b-7068-455b-9b49-2eec73e7a6ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:23.815383 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.815360    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/517c315b-7068-455b-9b49-2eec73e7a6ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "517c315b-7068-455b-9b49-2eec73e7a6ce" (UID: "517c315b-7068-455b-9b49-2eec73e7a6ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:32:23.815383 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.815369    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82214cf-4132-41b4-8807-84773974e8a6-kube-api-access-pf8w6" (OuterVolumeSpecName: "kube-api-access-pf8w6") pod "e82214cf-4132-41b4-8807-84773974e8a6" (UID: "e82214cf-4132-41b4-8807-84773974e8a6"). InnerVolumeSpecName "kube-api-access-pf8w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:32:23.815488 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.815397    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517c315b-7068-455b-9b49-2eec73e7a6ce-kube-api-access-jmr82" (OuterVolumeSpecName: "kube-api-access-jmr82") pod "517c315b-7068-455b-9b49-2eec73e7a6ce" (UID: "517c315b-7068-455b-9b49-2eec73e7a6ce"). InnerVolumeSpecName "kube-api-access-jmr82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:32:23.815488 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.815401    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9db3ebf-202e-4db5-93e7-f7c94407be15-kube-api-access-qjdhs" (OuterVolumeSpecName: "kube-api-access-qjdhs") pod "f9db3ebf-202e-4db5-93e7-f7c94407be15" (UID: "f9db3ebf-202e-4db5-93e7-f7c94407be15"). InnerVolumeSpecName "kube-api-access-qjdhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:32:23.860151 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.860067    2579 generic.go:356] "Generic (PLEG): container finished" podID="e82214cf-4132-41b4-8807-84773974e8a6" containerID="16dc200023054ee538bbfd167cf91b92ebec8e972b77a173a1b32cca5d908731" exitCode=0
Jul 29 19:32:23.860151 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.860123    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"
Jul 29 19:32:23.860396 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.860150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd" event={"ID":"e82214cf-4132-41b4-8807-84773974e8a6","Type":"ContainerDied","Data":"16dc200023054ee538bbfd167cf91b92ebec8e972b77a173a1b32cca5d908731"}
Jul 29 19:32:23.860396 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.860223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd" event={"ID":"e82214cf-4132-41b4-8807-84773974e8a6","Type":"ContainerDied","Data":"1ef47a68d232de5fe59d76dbe1497cd3ad5cf2f37a2be0d4df9b70fb3906b6c7"}
Jul 29 19:32:23.860396 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.860252    2579 scope.go:117] "RemoveContainer" containerID="16dc200023054ee538bbfd167cf91b92ebec8e972b77a173a1b32cca5d908731"
Jul 29 19:32:23.861521 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.861500    2579 generic.go:356] "Generic (PLEG): container finished" podID="f9db3ebf-202e-4db5-93e7-f7c94407be15" containerID="2c4f464546a2ecb709dc9bd8a9782c1ef16c29761dba7e5f17e861f27bffc7e3" exitCode=0
Jul 29 19:32:23.861650 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.861545    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"
Jul 29 19:32:23.861650 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.861581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb" event={"ID":"f9db3ebf-202e-4db5-93e7-f7c94407be15","Type":"ContainerDied","Data":"2c4f464546a2ecb709dc9bd8a9782c1ef16c29761dba7e5f17e861f27bffc7e3"}
Jul 29 19:32:23.861650 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.861637    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb" event={"ID":"f9db3ebf-202e-4db5-93e7-f7c94407be15","Type":"ContainerDied","Data":"85296d953983c6c178a1ba3ac6aa7083ca70ef630d336fc675a51add2c0d376d"}
Jul 29 19:32:23.863230 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.863177    2579 generic.go:356] "Generic (PLEG): container finished" podID="517c315b-7068-455b-9b49-2eec73e7a6ce" containerID="2b21e808ff03911dc35c460e01f654f93ad6bd12a4b6ce136fac8bd8f0c67514" exitCode=0
Jul 29 19:32:23.863351 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.863236    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"
Jul 29 19:32:23.863351 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.863234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb" event={"ID":"517c315b-7068-455b-9b49-2eec73e7a6ce","Type":"ContainerDied","Data":"2b21e808ff03911dc35c460e01f654f93ad6bd12a4b6ce136fac8bd8f0c67514"}
Jul 29 19:32:23.863351 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.863339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb" event={"ID":"517c315b-7068-455b-9b49-2eec73e7a6ce","Type":"ContainerDied","Data":"47c885bc4c6ba20bc6d0f1f7444b6426bfd5b3059aa7c9bc9336dbf36a886e1e"}
Jul 29 19:32:23.869781 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.869757    2579 scope.go:117] "RemoveContainer" containerID="16dc200023054ee538bbfd167cf91b92ebec8e972b77a173a1b32cca5d908731"
Jul 29 19:32:23.870089 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:32:23.870053    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16dc200023054ee538bbfd167cf91b92ebec8e972b77a173a1b32cca5d908731\": container with ID starting with 16dc200023054ee538bbfd167cf91b92ebec8e972b77a173a1b32cca5d908731 not found: ID does not exist" containerID="16dc200023054ee538bbfd167cf91b92ebec8e972b77a173a1b32cca5d908731"
Jul 29 19:32:23.870148 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.870082    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16dc200023054ee538bbfd167cf91b92ebec8e972b77a173a1b32cca5d908731"} err="failed to get container status \"16dc200023054ee538bbfd167cf91b92ebec8e972b77a173a1b32cca5d908731\": rpc error: code = NotFound desc = could not find container \"16dc200023054ee538bbfd167cf91b92ebec8e972b77a173a1b32cca5d908731\": container with ID starting with 16dc200023054ee538bbfd167cf91b92ebec8e972b77a173a1b32cca5d908731 not found: ID does not exist"
Jul 29 19:32:23.870148 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.870108    2579 scope.go:117] "RemoveContainer" containerID="2c4f464546a2ecb709dc9bd8a9782c1ef16c29761dba7e5f17e861f27bffc7e3"
Jul 29 19:32:23.879050 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.879028    2579 scope.go:117] "RemoveContainer" containerID="2c4f464546a2ecb709dc9bd8a9782c1ef16c29761dba7e5f17e861f27bffc7e3"
Jul 29 19:32:23.879351 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:32:23.879333    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c4f464546a2ecb709dc9bd8a9782c1ef16c29761dba7e5f17e861f27bffc7e3\": container with ID starting with 2c4f464546a2ecb709dc9bd8a9782c1ef16c29761dba7e5f17e861f27bffc7e3 not found: ID does not exist" containerID="2c4f464546a2ecb709dc9bd8a9782c1ef16c29761dba7e5f17e861f27bffc7e3"
Jul 29 19:32:23.879410 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.879359    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c4f464546a2ecb709dc9bd8a9782c1ef16c29761dba7e5f17e861f27bffc7e3"} err="failed to get container status \"2c4f464546a2ecb709dc9bd8a9782c1ef16c29761dba7e5f17e861f27bffc7e3\": rpc error: code = NotFound desc = could not find container \"2c4f464546a2ecb709dc9bd8a9782c1ef16c29761dba7e5f17e861f27bffc7e3\": container with ID starting with 2c4f464546a2ecb709dc9bd8a9782c1ef16c29761dba7e5f17e861f27bffc7e3 not found: ID does not exist"
Jul 29 19:32:23.879410 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.879376    2579 scope.go:117] "RemoveContainer" containerID="2b21e808ff03911dc35c460e01f654f93ad6bd12a4b6ce136fac8bd8f0c67514"
Jul 29 19:32:23.887635 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.887613    2579 scope.go:117] "RemoveContainer" containerID="2b21e808ff03911dc35c460e01f654f93ad6bd12a4b6ce136fac8bd8f0c67514"
Jul 29 19:32:23.887912 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:32:23.887893    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b21e808ff03911dc35c460e01f654f93ad6bd12a4b6ce136fac8bd8f0c67514\": container with ID starting with 2b21e808ff03911dc35c460e01f654f93ad6bd12a4b6ce136fac8bd8f0c67514 not found: ID does not exist" containerID="2b21e808ff03911dc35c460e01f654f93ad6bd12a4b6ce136fac8bd8f0c67514"
Jul 29 19:32:23.887985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.887924    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b21e808ff03911dc35c460e01f654f93ad6bd12a4b6ce136fac8bd8f0c67514"} err="failed to get container status \"2b21e808ff03911dc35c460e01f654f93ad6bd12a4b6ce136fac8bd8f0c67514\": rpc error: code = NotFound desc = could not find container \"2b21e808ff03911dc35c460e01f654f93ad6bd12a4b6ce136fac8bd8f0c67514\": container with ID starting with 2b21e808ff03911dc35c460e01f654f93ad6bd12a4b6ce136fac8bd8f0c67514 not found: ID does not exist"
Jul 29 19:32:23.887985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.887944    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"]
Jul 29 19:32:23.892092 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.892070    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a448180e-f59bd8596-fv5gd"]
Jul 29 19:32:23.904088 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.904064    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"]
Jul 29 19:32:23.908694 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.908674    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a448180e-68574b5bf4-x5kpb"]
Jul 29 19:32:23.914171 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.914149    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517c315b-7068-455b-9b49-2eec73e7a6ce-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:23.914253 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.914179    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517c315b-7068-455b-9b49-2eec73e7a6ce-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:23.914253 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.914220    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmr82\" (UniqueName: \"kubernetes.io/projected/517c315b-7068-455b-9b49-2eec73e7a6ce-kube-api-access-jmr82\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:23.914253 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.914235    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjdhs\" (UniqueName: \"kubernetes.io/projected/f9db3ebf-202e-4db5-93e7-f7c94407be15-kube-api-access-qjdhs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:23.914253 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.914249    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf8w6\" (UniqueName: \"kubernetes.io/projected/e82214cf-4132-41b4-8807-84773974e8a6-kube-api-access-pf8w6\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:23.914374 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.914262    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9db3ebf-202e-4db5-93e7-f7c94407be15-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:23.914374 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.914275    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82214cf-4132-41b4-8807-84773974e8a6-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:23.926444 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.926412    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"]
Jul 29 19:32:23.928501 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:23.928478    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a448180e-78d56765cb-cbtxb"]
Jul 29 19:32:24.201705 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:24.201674    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="517c315b-7068-455b-9b49-2eec73e7a6ce" path="/var/lib/kubelet/pods/517c315b-7068-455b-9b49-2eec73e7a6ce/volumes"
Jul 29 19:32:24.202002 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:24.201989    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72188e14-b439-4962-a359-66c339e047ef" path="/var/lib/kubelet/pods/72188e14-b439-4962-a359-66c339e047ef/volumes"
Jul 29 19:32:24.202306 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:24.202295    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e82214cf-4132-41b4-8807-84773974e8a6" path="/var/lib/kubelet/pods/e82214cf-4132-41b4-8807-84773974e8a6/volumes"
Jul 29 19:32:24.202607 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:24.202596    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9db3ebf-202e-4db5-93e7-f7c94407be15" path="/var/lib/kubelet/pods/f9db3ebf-202e-4db5-93e7-f7c94407be15/volumes"
Jul 29 19:32:27.096097 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.096059    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"]
Jul 29 19:32:27.096583 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.096566    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9db3ebf-202e-4db5-93e7-f7c94407be15" containerName="payload-processing"
Jul 29 19:32:27.096625 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.096586    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9db3ebf-202e-4db5-93e7-f7c94407be15" containerName="payload-processing"
Jul 29 19:32:27.096625 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.096599    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517c315b-7068-455b-9b49-2eec73e7a6ce" containerName="maas-api"
Jul 29 19:32:27.096625 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.096605    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="517c315b-7068-455b-9b49-2eec73e7a6ce" containerName="maas-api"
Jul 29 19:32:27.096625 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.096614    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e82214cf-4132-41b4-8807-84773974e8a6" containerName="payload-pre-processing"
Jul 29 19:32:27.096625 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.096620    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e82214cf-4132-41b4-8807-84773974e8a6" containerName="payload-pre-processing"
Jul 29 19:32:27.096770 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.096629    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72188e14-b439-4962-a359-66c339e047ef" containerName="revoke-keys"
Jul 29 19:32:27.096770 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.096635    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="72188e14-b439-4962-a359-66c339e047ef" containerName="revoke-keys"
Jul 29 19:32:27.096770 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.096692    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="72188e14-b439-4962-a359-66c339e047ef" containerName="revoke-keys"
Jul 29 19:32:27.096770 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.096700    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e82214cf-4132-41b4-8807-84773974e8a6" containerName="payload-pre-processing"
Jul 29 19:32:27.096770 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.096707    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="517c315b-7068-455b-9b49-2eec73e7a6ce" containerName="maas-api"
Jul 29 19:32:27.096770 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.096712    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9db3ebf-202e-4db5-93e7-f7c94407be15" containerName="payload-processing"
Jul 29 19:32:27.104798 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.104762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:27.107813 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.107788    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"]
Jul 29 19:32:27.108014 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.107948    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:32:27.108014 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.108005    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5bzmf\""
Jul 29 19:32:27.109826 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.109806    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e80635b4-serving-cert\""
Jul 29 19:32:27.109826 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.109819    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:32:27.246688 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.246648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-maas-api-tls\") pod \"maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz\" (UID: \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:27.246885 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.246732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-openshift-service-ca\") pod \"maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz\" (UID: \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:27.246885 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.246766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m749q\" (UniqueName: \"kubernetes.io/projected/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-kube-api-access-m749q\") pod \"maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz\" (UID: \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:27.347341 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.347251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-openshift-service-ca\") pod \"maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz\" (UID: \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:27.347341 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.347287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m749q\" (UniqueName: \"kubernetes.io/projected/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-kube-api-access-m749q\") pod \"maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz\" (UID: \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:27.347550 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.347367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-maas-api-tls\") pod \"maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz\" (UID: \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:27.347550 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:32:27.347482    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-serving-cert: secret "maas-api-e2e-migrate-e80635b4-serving-cert" not found
Jul 29 19:32:27.347624 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:32:27.347557    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-maas-api-tls podName:1b092b6b-089a-4ff3-a2f2-89f6d7572e20 nodeName:}" failed. No retries permitted until 2026-07-29 19:32:27.84753467 +0000 UTC m=+1914.147207150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-maas-api-tls") pod "maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz" (UID: "1b092b6b-089a-4ff3-a2f2-89f6d7572e20") : secret "maas-api-e2e-migrate-e80635b4-serving-cert" not found
Jul 29 19:32:27.348003 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.347984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-openshift-service-ca\") pod \"maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz\" (UID: \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:27.357062 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.357035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m749q\" (UniqueName: \"kubernetes.io/projected/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-kube-api-access-m749q\") pod \"maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz\" (UID: \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:27.733521 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.733484    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"]
Jul 29 19:32:27.737620 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.737602    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"
Jul 29 19:32:27.740558 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.740537    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e80635b4-dockercfg-pt4lp\""
Jul 29 19:32:27.740676 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.740595    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e80635b4\""
Jul 29 19:32:27.744322 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.744296    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"]
Jul 29 19:32:27.754847 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.754820    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"]
Jul 29 19:32:27.758607 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.758591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"
Jul 29 19:32:27.772705 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.772677    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"]
Jul 29 19:32:27.851290 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.851244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f23ed33-601f-4618-8c41-ae84d789de8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd\" (UID: \"9f23ed33-601f-4618-8c41-ae84d789de8d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"
Jul 29 19:32:27.851467 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.851305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e1e575-352a-42e6-a178-a0979ad85a74-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw\" (UID: \"80e1e575-352a-42e6-a178-a0979ad85a74\") " pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"
Jul 29 19:32:27.851467 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.851366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-maas-api-tls\") pod \"maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz\" (UID: \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:27.851467 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.851415    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsbw6\" (UniqueName: \"kubernetes.io/projected/9f23ed33-601f-4618-8c41-ae84d789de8d-kube-api-access-qsbw6\") pod \"payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd\" (UID: \"9f23ed33-601f-4618-8c41-ae84d789de8d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"
Jul 29 19:32:27.851632 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.851508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85djf\" (UniqueName: \"kubernetes.io/projected/80e1e575-352a-42e6-a178-a0979ad85a74-kube-api-access-85djf\") pod \"payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw\" (UID: \"80e1e575-352a-42e6-a178-a0979ad85a74\") " pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"
Jul 29 19:32:27.853850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.853834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-maas-api-tls\") pod \"maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz\" (UID: \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:27.952254 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.952187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f23ed33-601f-4618-8c41-ae84d789de8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd\" (UID: \"9f23ed33-601f-4618-8c41-ae84d789de8d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"
Jul 29 19:32:27.952513 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.952271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e1e575-352a-42e6-a178-a0979ad85a74-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw\" (UID: \"80e1e575-352a-42e6-a178-a0979ad85a74\") " pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"
Jul 29 19:32:27.952513 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.952321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsbw6\" (UniqueName: \"kubernetes.io/projected/9f23ed33-601f-4618-8c41-ae84d789de8d-kube-api-access-qsbw6\") pod \"payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd\" (UID: \"9f23ed33-601f-4618-8c41-ae84d789de8d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"
Jul 29 19:32:27.952513 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.952443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85djf\" (UniqueName: \"kubernetes.io/projected/80e1e575-352a-42e6-a178-a0979ad85a74-kube-api-access-85djf\") pod \"payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw\" (UID: \"80e1e575-352a-42e6-a178-a0979ad85a74\") " pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"
Jul 29 19:32:27.952983 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.952967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f23ed33-601f-4618-8c41-ae84d789de8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd\" (UID: \"9f23ed33-601f-4618-8c41-ae84d789de8d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"
Jul 29 19:32:27.953020 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.952993    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e1e575-352a-42e6-a178-a0979ad85a74-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw\" (UID: \"80e1e575-352a-42e6-a178-a0979ad85a74\") " pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"
Jul 29 19:32:27.961736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.961714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85djf\" (UniqueName: \"kubernetes.io/projected/80e1e575-352a-42e6-a178-a0979ad85a74-kube-api-access-85djf\") pod \"payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw\" (UID: \"80e1e575-352a-42e6-a178-a0979ad85a74\") " pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"
Jul 29 19:32:27.961884 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:27.961761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsbw6\" (UniqueName: \"kubernetes.io/projected/9f23ed33-601f-4618-8c41-ae84d789de8d-kube-api-access-qsbw6\") pod \"payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd\" (UID: \"9f23ed33-601f-4618-8c41-ae84d789de8d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"
Jul 29 19:32:28.018164 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.018080    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:28.049931 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.049889    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"
Jul 29 19:32:28.069708 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.069678    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"
Jul 29 19:32:28.221382 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.218978    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"]
Jul 29 19:32:28.234111 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:32:28.232954    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b092b6b_089a_4ff3_a2f2_89f6d7572e20.slice/crio-265ac119b3c88577dbbc6b4f01a8d73a25437380d67f0e72fe4fa8b0a778ca68 WatchSource:0}: Error finding container 265ac119b3c88577dbbc6b4f01a8d73a25437380d67f0e72fe4fa8b0a778ca68: Status 404 returned error can't find the container with id 265ac119b3c88577dbbc6b4f01a8d73a25437380d67f0e72fe4fa8b0a778ca68
Jul 29 19:32:28.252831 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.252802    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"]
Jul 29 19:32:28.255326 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:32:28.255300    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f23ed33_601f_4618_8c41_ae84d789de8d.slice/crio-50588df18d4dda0474db94791d9433d9bc0445bac0b0bfb13a77abd4eb18dd01 WatchSource:0}: Error finding container 50588df18d4dda0474db94791d9433d9bc0445bac0b0bfb13a77abd4eb18dd01: Status 404 returned error can't find the container with id 50588df18d4dda0474db94791d9433d9bc0445bac0b0bfb13a77abd4eb18dd01
Jul 29 19:32:28.281581 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.281557    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"]
Jul 29 19:32:28.293460 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:32:28.293434    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e1e575_352a_42e6_a178_a0979ad85a74.slice/crio-51bf464391edd33e76e9c14f166188b783916a9677fb81ce9ddcf3c5bbb1a4a8 WatchSource:0}: Error finding container 51bf464391edd33e76e9c14f166188b783916a9677fb81ce9ddcf3c5bbb1a4a8: Status 404 returned error can't find the container with id 51bf464391edd33e76e9c14f166188b783916a9677fb81ce9ddcf3c5bbb1a4a8
Jul 29 19:32:28.894103 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.894061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz" event={"ID":"1b092b6b-089a-4ff3-a2f2-89f6d7572e20","Type":"ContainerStarted","Data":"03b8fbfaec4e96a4d4a6be21a25f27492d18138da9ad2663b4302eaac7385e65"}
Jul 29 19:32:28.894363 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.894127    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz" event={"ID":"1b092b6b-089a-4ff3-a2f2-89f6d7572e20","Type":"ContainerStarted","Data":"265ac119b3c88577dbbc6b4f01a8d73a25437380d67f0e72fe4fa8b0a778ca68"}
Jul 29 19:32:28.894363 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.894173    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:28.896211 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.896169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd" event={"ID":"9f23ed33-601f-4618-8c41-ae84d789de8d","Type":"ContainerStarted","Data":"2db43d42d31654a28f41e3951d5dc78061b08ef8bbb864347f370a554d41412e"}
Jul 29 19:32:28.896341 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.896248    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"
Jul 29 19:32:28.896341 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.896274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd" event={"ID":"9f23ed33-601f-4618-8c41-ae84d789de8d","Type":"ContainerStarted","Data":"50588df18d4dda0474db94791d9433d9bc0445bac0b0bfb13a77abd4eb18dd01"}
Jul 29 19:32:28.897901 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.897873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw" event={"ID":"80e1e575-352a-42e6-a178-a0979ad85a74","Type":"ContainerStarted","Data":"71848e26b2dd801222bea15fa2c4134043b44fc26e8f480c50f8771cfe1b9b5f"}
Jul 29 19:32:28.898010 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.897906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw" event={"ID":"80e1e575-352a-42e6-a178-a0979ad85a74","Type":"ContainerStarted","Data":"51bf464391edd33e76e9c14f166188b783916a9677fb81ce9ddcf3c5bbb1a4a8"}
Jul 29 19:32:28.898070 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.898041    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"
Jul 29 19:32:28.925268 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.925153    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz" podStartSLOduration=1.4981886260000001 podStartE2EDuration="1.925137889s" totalImagesPullingTime="426.949263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:32:28.923932751 +0000 UTC m=+1915.223605252" watchObservedRunningTime="2026-07-29 19:32:28.925137889 +0000 UTC m=+1915.224810388"
Jul 29 19:32:28.945962 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.945914    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd" podStartSLOduration=1.9458983970000001 podStartE2EDuration="1.945898397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:32:28.94461275 +0000 UTC m=+1915.244285251" watchObservedRunningTime="2026-07-29 19:32:28.945898397 +0000 UTC m=+1915.245570896"
Jul 29 19:32:28.962729 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:28.962680    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw" podStartSLOduration=1.962668474 podStartE2EDuration="1.962668474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:32:28.961449112 +0000 UTC m=+1915.261121614" watchObservedRunningTime="2026-07-29 19:32:28.962668474 +0000 UTC m=+1915.262340974"
Jul 29 19:32:34.909377 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:34.909348    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:39.906120 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:39.906090    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"
Jul 29 19:32:39.906633 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:39.906156    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"
Jul 29 19:32:51.991619 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:51.991580    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn"]
Jul 29 19:32:51.995346 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:51.995330    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn"
Jul 29 19:32:51.998442 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:51.998415    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq9q2\""
Jul 29 19:32:52.005298 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:52.005269    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn"]
Jul 29 19:32:52.069624 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:52.069588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/297c5366-1441-46dc-9b97-f106b65ecb82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn\" (UID: \"297c5366-1441-46dc-9b97-f106b65ecb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn"
Jul 29 19:32:52.171113 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:52.171065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/297c5366-1441-46dc-9b97-f106b65ecb82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn\" (UID: \"297c5366-1441-46dc-9b97-f106b65ecb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn"
Jul 29 19:32:52.172002 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:52.171953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/297c5366-1441-46dc-9b97-f106b65ecb82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn\" (UID: \"297c5366-1441-46dc-9b97-f106b65ecb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn"
Jul 29 19:32:52.306711 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:52.306599    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn"
Jul 29 19:32:52.444979 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:52.444951    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn"]
Jul 29 19:32:52.446929 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:32:52.446902    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297c5366_1441_46dc_9b97_f106b65ecb82.slice/crio-57eb2445cc14736bdadfedfa806fd1dda90c90d8dafc59166b735c8bd2c29c0d WatchSource:0}: Error finding container 57eb2445cc14736bdadfedfa806fd1dda90c90d8dafc59166b735c8bd2c29c0d: Status 404 returned error can't find the container with id 57eb2445cc14736bdadfedfa806fd1dda90c90d8dafc59166b735c8bd2c29c0d
Jul 29 19:32:53.021725 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:53.021694    2579 generic.go:356] "Generic (PLEG): container finished" podID="297c5366-1441-46dc-9b97-f106b65ecb82" containerID="dcef783d7a47b546a9f0cc6944e927f9a2403c45e0c1d26a9b2d056028a8b88e" exitCode=0
Jul 29 19:32:53.022058 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:53.021774    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn" event={"ID":"297c5366-1441-46dc-9b97-f106b65ecb82","Type":"ContainerDied","Data":"dcef783d7a47b546a9f0cc6944e927f9a2403c45e0c1d26a9b2d056028a8b88e"}
Jul 29 19:32:53.022058 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:53.021832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn" event={"ID":"297c5366-1441-46dc-9b97-f106b65ecb82","Type":"ContainerStarted","Data":"57eb2445cc14736bdadfedfa806fd1dda90c90d8dafc59166b735c8bd2c29c0d"}
Jul 29 19:32:54.154625 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:54.154601    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn"
Jul 29 19:32:54.290546 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:54.290450    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/297c5366-1441-46dc-9b97-f106b65ecb82-maas-api-service-ca\") pod \"297c5366-1441-46dc-9b97-f106b65ecb82\" (UID: \"297c5366-1441-46dc-9b97-f106b65ecb82\") "
Jul 29 19:32:54.290886 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:54.290850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297c5366-1441-46dc-9b97-f106b65ecb82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "297c5366-1441-46dc-9b97-f106b65ecb82" (UID: "297c5366-1441-46dc-9b97-f106b65ecb82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:54.391798 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:54.391761    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/297c5366-1441-46dc-9b97-f106b65ecb82-maas-api-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:55.032750 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:55.032724    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn"
Jul 29 19:32:55.032926 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:55.032765    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn" event={"ID":"297c5366-1441-46dc-9b97-f106b65ecb82","Type":"ContainerDied","Data":"57eb2445cc14736bdadfedfa806fd1dda90c90d8dafc59166b735c8bd2c29c0d"}
Jul 29 19:32:55.032926 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:55.032813    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57eb2445cc14736bdadfedfa806fd1dda90c90d8dafc59166b735c8bd2c29c0d"
Jul 29 19:32:57.769470 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:57.769428    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"]
Jul 29 19:32:57.770067 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:57.769643    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz" podUID="1b092b6b-089a-4ff3-a2f2-89f6d7572e20" containerName="maas-api" containerID="cri-o://03b8fbfaec4e96a4d4a6be21a25f27492d18138da9ad2663b4302eaac7385e65" gracePeriod=30
Jul 29 19:32:57.828405 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:57.828375    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"]
Jul 29 19:32:57.828615 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:57.828586    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw" podUID="80e1e575-352a-42e6-a178-a0979ad85a74" containerName="payload-processing" containerID="cri-o://71848e26b2dd801222bea15fa2c4134043b44fc26e8f480c50f8771cfe1b9b5f" gracePeriod=30
Jul 29 19:32:57.842499 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:57.842465    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"]
Jul 29 19:32:57.843044 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:57.843013    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd" podUID="9f23ed33-601f-4618-8c41-ae84d789de8d" containerName="payload-pre-processing" containerID="cri-o://2db43d42d31654a28f41e3951d5dc78061b08ef8bbb864347f370a554d41412e" gracePeriod=30
Jul 29 19:32:57.971555 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:57.971518    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn"]
Jul 29 19:32:57.973537 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:57.973500    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e80635b4-xdpsn"]
Jul 29 19:32:58.050300 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.050267    2579 generic.go:356] "Generic (PLEG): container finished" podID="9f23ed33-601f-4618-8c41-ae84d789de8d" containerID="2db43d42d31654a28f41e3951d5dc78061b08ef8bbb864347f370a554d41412e" exitCode=0
Jul 29 19:32:58.050462 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.050384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd" event={"ID":"9f23ed33-601f-4618-8c41-ae84d789de8d","Type":"ContainerDied","Data":"2db43d42d31654a28f41e3951d5dc78061b08ef8bbb864347f370a554d41412e"}
Jul 29 19:32:58.052163 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.052139    2579 generic.go:356] "Generic (PLEG): container finished" podID="80e1e575-352a-42e6-a178-a0979ad85a74" containerID="71848e26b2dd801222bea15fa2c4134043b44fc26e8f480c50f8771cfe1b9b5f" exitCode=0
Jul 29 19:32:58.052316 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.052230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw" event={"ID":"80e1e575-352a-42e6-a178-a0979ad85a74","Type":"ContainerDied","Data":"71848e26b2dd801222bea15fa2c4134043b44fc26e8f480c50f8771cfe1b9b5f"}
Jul 29 19:32:58.054007 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.053979    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b092b6b-089a-4ff3-a2f2-89f6d7572e20" containerID="03b8fbfaec4e96a4d4a6be21a25f27492d18138da9ad2663b4302eaac7385e65" exitCode=0
Jul 29 19:32:58.054119 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.054030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz" event={"ID":"1b092b6b-089a-4ff3-a2f2-89f6d7572e20","Type":"ContainerDied","Data":"03b8fbfaec4e96a4d4a6be21a25f27492d18138da9ad2663b4302eaac7385e65"}
Jul 29 19:32:58.120558 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.120531    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"
Jul 29 19:32:58.133635 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.133605    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"
Jul 29 19:32:58.141767 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.141740    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:58.196176 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.196143    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="297c5366-1441-46dc-9b97-f106b65ecb82" path="/var/lib/kubelet/pods/297c5366-1441-46dc-9b97-f106b65ecb82/volumes"
Jul 29 19:32:58.230641 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.230610    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m749q\" (UniqueName: \"kubernetes.io/projected/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-kube-api-access-m749q\") pod \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\" (UID: \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\") "
Jul 29 19:32:58.230808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.230685    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e1e575-352a-42e6-a178-a0979ad85a74-plugins-config-volume\") pod \"80e1e575-352a-42e6-a178-a0979ad85a74\" (UID: \"80e1e575-352a-42e6-a178-a0979ad85a74\") "
Jul 29 19:32:58.230808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.230715    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-openshift-service-ca\") pod \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\" (UID: \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\") "
Jul 29 19:32:58.230808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.230734    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85djf\" (UniqueName: \"kubernetes.io/projected/80e1e575-352a-42e6-a178-a0979ad85a74-kube-api-access-85djf\") pod \"80e1e575-352a-42e6-a178-a0979ad85a74\" (UID: \"80e1e575-352a-42e6-a178-a0979ad85a74\") "
Jul 29 19:32:58.230808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.230763    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f23ed33-601f-4618-8c41-ae84d789de8d-plugins-config-volume\") pod \"9f23ed33-601f-4618-8c41-ae84d789de8d\" (UID: \"9f23ed33-601f-4618-8c41-ae84d789de8d\") "
Jul 29 19:32:58.230808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.230783    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-maas-api-tls\") pod \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\" (UID: \"1b092b6b-089a-4ff3-a2f2-89f6d7572e20\") "
Jul 29 19:32:58.231073 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.230850    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsbw6\" (UniqueName: \"kubernetes.io/projected/9f23ed33-601f-4618-8c41-ae84d789de8d-kube-api-access-qsbw6\") pod \"9f23ed33-601f-4618-8c41-ae84d789de8d\" (UID: \"9f23ed33-601f-4618-8c41-ae84d789de8d\") "
Jul 29 19:32:58.231139 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.231114    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b092b6b-089a-4ff3-a2f2-89f6d7572e20" (UID: "1b092b6b-089a-4ff3-a2f2-89f6d7572e20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:58.231212 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.231138    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e1e575-352a-42e6-a178-a0979ad85a74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80e1e575-352a-42e6-a178-a0979ad85a74" (UID: "80e1e575-352a-42e6-a178-a0979ad85a74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:58.231212 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.231166    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f23ed33-601f-4618-8c41-ae84d789de8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f23ed33-601f-4618-8c41-ae84d789de8d" (UID: "9f23ed33-601f-4618-8c41-ae84d789de8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:58.233022 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.232992    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b092b6b-089a-4ff3-a2f2-89f6d7572e20" (UID: "1b092b6b-089a-4ff3-a2f2-89f6d7572e20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:32:58.233140 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.233025    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-kube-api-access-m749q" (OuterVolumeSpecName: "kube-api-access-m749q") pod "1b092b6b-089a-4ff3-a2f2-89f6d7572e20" (UID: "1b092b6b-089a-4ff3-a2f2-89f6d7572e20"). InnerVolumeSpecName "kube-api-access-m749q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:32:58.233140 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.233032    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e1e575-352a-42e6-a178-a0979ad85a74-kube-api-access-85djf" (OuterVolumeSpecName: "kube-api-access-85djf") pod "80e1e575-352a-42e6-a178-a0979ad85a74" (UID: "80e1e575-352a-42e6-a178-a0979ad85a74"). InnerVolumeSpecName "kube-api-access-85djf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:32:58.233140 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.233065    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f23ed33-601f-4618-8c41-ae84d789de8d-kube-api-access-qsbw6" (OuterVolumeSpecName: "kube-api-access-qsbw6") pod "9f23ed33-601f-4618-8c41-ae84d789de8d" (UID: "9f23ed33-601f-4618-8c41-ae84d789de8d"). InnerVolumeSpecName "kube-api-access-qsbw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:32:58.332101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.331991    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f23ed33-601f-4618-8c41-ae84d789de8d-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:58.332101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.332065    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:58.332101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.332083    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsbw6\" (UniqueName: \"kubernetes.io/projected/9f23ed33-601f-4618-8c41-ae84d789de8d-kube-api-access-qsbw6\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:58.332101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.332096    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m749q\" (UniqueName: \"kubernetes.io/projected/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-kube-api-access-m749q\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:58.332101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.332110    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e1e575-352a-42e6-a178-a0979ad85a74-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:58.332436 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.332122    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b092b6b-089a-4ff3-a2f2-89f6d7572e20-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:58.332436 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:58.332134    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85djf\" (UniqueName: \"kubernetes.io/projected/80e1e575-352a-42e6-a178-a0979ad85a74-kube-api-access-85djf\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:32:59.060315 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.060223    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"
Jul 29 19:32:59.060315 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.060236    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd" event={"ID":"9f23ed33-601f-4618-8c41-ae84d789de8d","Type":"ContainerDied","Data":"50588df18d4dda0474db94791d9433d9bc0445bac0b0bfb13a77abd4eb18dd01"}
Jul 29 19:32:59.060315 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.060304    2579 scope.go:117] "RemoveContainer" containerID="2db43d42d31654a28f41e3951d5dc78061b08ef8bbb864347f370a554d41412e"
Jul 29 19:32:59.061592 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.061528    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"
Jul 29 19:32:59.061592 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.061570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw" event={"ID":"80e1e575-352a-42e6-a178-a0979ad85a74","Type":"ContainerDied","Data":"51bf464391edd33e76e9c14f166188b783916a9677fb81ce9ddcf3c5bbb1a4a8"}
Jul 29 19:32:59.063296 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.063269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz" event={"ID":"1b092b6b-089a-4ff3-a2f2-89f6d7572e20","Type":"ContainerDied","Data":"265ac119b3c88577dbbc6b4f01a8d73a25437380d67f0e72fe4fa8b0a778ca68"}
Jul 29 19:32:59.063418 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.063302    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"
Jul 29 19:32:59.070794 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.070753    2579 scope.go:117] "RemoveContainer" containerID="71848e26b2dd801222bea15fa2c4134043b44fc26e8f480c50f8771cfe1b9b5f"
Jul 29 19:32:59.079594 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.079575    2579 scope.go:117] "RemoveContainer" containerID="03b8fbfaec4e96a4d4a6be21a25f27492d18138da9ad2663b4302eaac7385e65"
Jul 29 19:32:59.090134 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.090110    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"]
Jul 29 19:32:59.091779 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.091760    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e80635b4-6c7bccd888-b54qd"]
Jul 29 19:32:59.103763 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.103732    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"]
Jul 29 19:32:59.105384 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.105362    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e80635b4-5f8887f9bf-bc6sw"]
Jul 29 19:32:59.116641 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.116608    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"]
Jul 29 19:32:59.127066 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:32:59.127030    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e80635b4-94b4cf7b7-dmwvz"]
Jul 29 19:33:00.196348 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:00.196316    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b092b6b-089a-4ff3-a2f2-89f6d7572e20" path="/var/lib/kubelet/pods/1b092b6b-089a-4ff3-a2f2-89f6d7572e20/volumes"
Jul 29 19:33:00.196719 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:00.196638    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80e1e575-352a-42e6-a178-a0979ad85a74" path="/var/lib/kubelet/pods/80e1e575-352a-42e6-a178-a0979ad85a74/volumes"
Jul 29 19:33:00.196932 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:00.196919    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f23ed33-601f-4618-8c41-ae84d789de8d" path="/var/lib/kubelet/pods/9f23ed33-601f-4618-8c41-ae84d789de8d/volumes"
Jul 29 19:33:01.233273 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.233238    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"]
Jul 29 19:33:01.233664 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.233625    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="297c5366-1441-46dc-9b97-f106b65ecb82" containerName="revoke-keys"
Jul 29 19:33:01.233664 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.233636    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="297c5366-1441-46dc-9b97-f106b65ecb82" containerName="revoke-keys"
Jul 29 19:33:01.233664 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.233647    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f23ed33-601f-4618-8c41-ae84d789de8d" containerName="payload-pre-processing"
Jul 29 19:33:01.233664 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.233653    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f23ed33-601f-4618-8c41-ae84d789de8d" containerName="payload-pre-processing"
Jul 29 19:33:01.233664 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.233662    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b092b6b-089a-4ff3-a2f2-89f6d7572e20" containerName="maas-api"
Jul 29 19:33:01.233664 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.233668    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b092b6b-089a-4ff3-a2f2-89f6d7572e20" containerName="maas-api"
Jul 29 19:33:01.233881 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.233682    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e1e575-352a-42e6-a178-a0979ad85a74" containerName="payload-processing"
Jul 29 19:33:01.233881 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.233687    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e1e575-352a-42e6-a178-a0979ad85a74" containerName="payload-processing"
Jul 29 19:33:01.233881 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.233756    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b092b6b-089a-4ff3-a2f2-89f6d7572e20" containerName="maas-api"
Jul 29 19:33:01.233881 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.233769    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f23ed33-601f-4618-8c41-ae84d789de8d" containerName="payload-pre-processing"
Jul 29 19:33:01.233881 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.233774    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e1e575-352a-42e6-a178-a0979ad85a74" containerName="payload-processing"
Jul 29 19:33:01.233881 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.233780    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="297c5366-1441-46dc-9b97-f106b65ecb82" containerName="revoke-keys"
Jul 29 19:33:01.238473 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.238448    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:01.242295 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.242272    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6d42a5b6-serving-cert\""
Jul 29 19:33:01.243629 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.243611    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5bzmf\""
Jul 29 19:33:01.243629 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.243622    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:33:01.243766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.243671    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:33:01.251950 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.251929    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"]
Jul 29 19:33:01.361964 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.361930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f6c3c1-30dd-4ea1-93c2-682165982db1-openshift-service-ca\") pod \"maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd\" (UID: \"22f6c3c1-30dd-4ea1-93c2-682165982db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:01.362127 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.361998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22f6c3c1-30dd-4ea1-93c2-682165982db1-maas-api-tls\") pod \"maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd\" (UID: \"22f6c3c1-30dd-4ea1-93c2-682165982db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:01.362127 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.362038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl7gl\" (UniqueName: \"kubernetes.io/projected/22f6c3c1-30dd-4ea1-93c2-682165982db1-kube-api-access-pl7gl\") pod \"maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd\" (UID: \"22f6c3c1-30dd-4ea1-93c2-682165982db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:01.463295 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.463256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22f6c3c1-30dd-4ea1-93c2-682165982db1-maas-api-tls\") pod \"maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd\" (UID: \"22f6c3c1-30dd-4ea1-93c2-682165982db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:01.463508 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.463304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl7gl\" (UniqueName: \"kubernetes.io/projected/22f6c3c1-30dd-4ea1-93c2-682165982db1-kube-api-access-pl7gl\") pod \"maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd\" (UID: \"22f6c3c1-30dd-4ea1-93c2-682165982db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:01.463508 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.463369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f6c3c1-30dd-4ea1-93c2-682165982db1-openshift-service-ca\") pod \"maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd\" (UID: \"22f6c3c1-30dd-4ea1-93c2-682165982db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:01.464027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.464008    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f6c3c1-30dd-4ea1-93c2-682165982db1-openshift-service-ca\") pod \"maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd\" (UID: \"22f6c3c1-30dd-4ea1-93c2-682165982db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:01.465911 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.465872    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22f6c3c1-30dd-4ea1-93c2-682165982db1-maas-api-tls\") pod \"maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd\" (UID: \"22f6c3c1-30dd-4ea1-93c2-682165982db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:01.475705 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.475681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl7gl\" (UniqueName: \"kubernetes.io/projected/22f6c3c1-30dd-4ea1-93c2-682165982db1-kube-api-access-pl7gl\") pod \"maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd\" (UID: \"22f6c3c1-30dd-4ea1-93c2-682165982db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:01.550927 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.550806    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:01.702342 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:01.702308    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"]
Jul 29 19:33:01.703905 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:33:01.703876    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f6c3c1_30dd_4ea1_93c2_682165982db1.slice/crio-b083c99c3cc346ae3d11f279c86d905a3a76e80f3a8da159638b30a14994bedb WatchSource:0}: Error finding container b083c99c3cc346ae3d11f279c86d905a3a76e80f3a8da159638b30a14994bedb: Status 404 returned error can't find the container with id b083c99c3cc346ae3d11f279c86d905a3a76e80f3a8da159638b30a14994bedb
Jul 29 19:33:02.081724 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.081701    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd" event={"ID":"22f6c3c1-30dd-4ea1-93c2-682165982db1","Type":"ContainerStarted","Data":"b083c99c3cc346ae3d11f279c86d905a3a76e80f3a8da159638b30a14994bedb"}
Jul 29 19:33:02.151734 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.151665    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"]
Jul 29 19:33:02.155368 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.155345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"
Jul 29 19:33:02.158740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.158722    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6d42a5b6-dockercfg-lvpwc\""
Jul 29 19:33:02.158867 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.158725    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6d42a5b6\""
Jul 29 19:33:02.173906 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.173878    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"]
Jul 29 19:33:02.197458 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.197426    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"]
Jul 29 19:33:02.201567 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.201548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"
Jul 29 19:33:02.224783 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.224752    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"]
Jul 29 19:33:02.269625 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.269591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqjr\" (UniqueName: \"kubernetes.io/projected/a9284f51-a1c5-4f11-a6bd-05cc9b0aff56-kube-api-access-ztqjr\") pod \"payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p\" (UID: \"a9284f51-a1c5-4f11-a6bd-05cc9b0aff56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"
Jul 29 19:33:02.269625 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.269628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fe946e-a1a6-42c8-bc32-77d2060d7288-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb\" (UID: \"18fe946e-a1a6-42c8-bc32-77d2060d7288\") " pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"
Jul 29 19:33:02.270034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.269735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhnjw\" (UniqueName: \"kubernetes.io/projected/18fe946e-a1a6-42c8-bc32-77d2060d7288-kube-api-access-fhnjw\") pod \"payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb\" (UID: \"18fe946e-a1a6-42c8-bc32-77d2060d7288\") " pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"
Jul 29 19:33:02.270034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.269760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9284f51-a1c5-4f11-a6bd-05cc9b0aff56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p\" (UID: \"a9284f51-a1c5-4f11-a6bd-05cc9b0aff56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"
Jul 29 19:33:02.370881 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.370794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqjr\" (UniqueName: \"kubernetes.io/projected/a9284f51-a1c5-4f11-a6bd-05cc9b0aff56-kube-api-access-ztqjr\") pod \"payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p\" (UID: \"a9284f51-a1c5-4f11-a6bd-05cc9b0aff56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"
Jul 29 19:33:02.370881 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.370828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fe946e-a1a6-42c8-bc32-77d2060d7288-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb\" (UID: \"18fe946e-a1a6-42c8-bc32-77d2060d7288\") " pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"
Jul 29 19:33:02.371073 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.370883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhnjw\" (UniqueName: \"kubernetes.io/projected/18fe946e-a1a6-42c8-bc32-77d2060d7288-kube-api-access-fhnjw\") pod \"payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb\" (UID: \"18fe946e-a1a6-42c8-bc32-77d2060d7288\") " pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"
Jul 29 19:33:02.371073 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.370905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9284f51-a1c5-4f11-a6bd-05cc9b0aff56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p\" (UID: \"a9284f51-a1c5-4f11-a6bd-05cc9b0aff56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"
Jul 29 19:33:02.371565 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.371547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9284f51-a1c5-4f11-a6bd-05cc9b0aff56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p\" (UID: \"a9284f51-a1c5-4f11-a6bd-05cc9b0aff56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"
Jul 29 19:33:02.371610 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.371573    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fe946e-a1a6-42c8-bc32-77d2060d7288-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb\" (UID: \"18fe946e-a1a6-42c8-bc32-77d2060d7288\") " pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"
Jul 29 19:33:02.383934 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.383906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhnjw\" (UniqueName: \"kubernetes.io/projected/18fe946e-a1a6-42c8-bc32-77d2060d7288-kube-api-access-fhnjw\") pod \"payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb\" (UID: \"18fe946e-a1a6-42c8-bc32-77d2060d7288\") " pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"
Jul 29 19:33:02.384054 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.383906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqjr\" (UniqueName: \"kubernetes.io/projected/a9284f51-a1c5-4f11-a6bd-05cc9b0aff56-kube-api-access-ztqjr\") pod \"payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p\" (UID: \"a9284f51-a1c5-4f11-a6bd-05cc9b0aff56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"
Jul 29 19:33:02.465537 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.465498    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"
Jul 29 19:33:02.511351 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.511319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"
Jul 29 19:33:02.617992 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.617967    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"]
Jul 29 19:33:02.620058 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:33:02.620015    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fe946e_a1a6_42c8_bc32_77d2060d7288.slice/crio-9a58c6e2e35abb7568f243309574c6a3304e4c2638ef689f9f0f49c93e9ff518 WatchSource:0}: Error finding container 9a58c6e2e35abb7568f243309574c6a3304e4c2638ef689f9f0f49c93e9ff518: Status 404 returned error can't find the container with id 9a58c6e2e35abb7568f243309574c6a3304e4c2638ef689f9f0f49c93e9ff518
Jul 29 19:33:02.666282 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:02.666258    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"]
Jul 29 19:33:02.668157 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:33:02.668132    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9284f51_a1c5_4f11_a6bd_05cc9b0aff56.slice/crio-826d65dfe9359545af539b07ee58280bb67aa793437d071ed540d595f06e76bb WatchSource:0}: Error finding container 826d65dfe9359545af539b07ee58280bb67aa793437d071ed540d595f06e76bb: Status 404 returned error can't find the container with id 826d65dfe9359545af539b07ee58280bb67aa793437d071ed540d595f06e76bb
Jul 29 19:33:03.089617 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:03.089576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd" event={"ID":"22f6c3c1-30dd-4ea1-93c2-682165982db1","Type":"ContainerStarted","Data":"55febe1ee5117467c5a584fb4e04a2bbd19d10d59496521be1d2f9eae3e4fea4"}
Jul 29 19:33:03.089826 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:03.089647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:03.091290 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:03.091265    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p" event={"ID":"a9284f51-a1c5-4f11-a6bd-05cc9b0aff56","Type":"ContainerStarted","Data":"8327d990763c3aafd3333bd053ecc0bb73348c781debde8c4488ad0da2d1257a"}
Jul 29 19:33:03.091417 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:03.091317    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p" event={"ID":"a9284f51-a1c5-4f11-a6bd-05cc9b0aff56","Type":"ContainerStarted","Data":"826d65dfe9359545af539b07ee58280bb67aa793437d071ed540d595f06e76bb"}
Jul 29 19:33:03.091417 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:03.091346    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"
Jul 29 19:33:03.092962 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:03.092939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb" event={"ID":"18fe946e-a1a6-42c8-bc32-77d2060d7288","Type":"ContainerStarted","Data":"9170def1432ee01f03affd742b43035804b89381e81a14f2effcb69431e174a7"}
Jul 29 19:33:03.093065 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:03.092982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb" event={"ID":"18fe946e-a1a6-42c8-bc32-77d2060d7288","Type":"ContainerStarted","Data":"9a58c6e2e35abb7568f243309574c6a3304e4c2638ef689f9f0f49c93e9ff518"}
Jul 29 19:33:03.093065 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:03.093036    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"
Jul 29 19:33:03.114395 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:03.114345    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd" podStartSLOduration=1.740277776 podStartE2EDuration="2.1143282s" totalImagesPullingTime="374.050424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:33:03.114069179 +0000 UTC m=+1949.413741674" watchObservedRunningTime="2026-07-29 19:33:03.1143282 +0000 UTC m=+1949.414000700"
Jul 29 19:33:03.134786 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:03.134735    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p" podStartSLOduration=1.134719196 podStartE2EDuration="1.134719196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:33:03.133555295 +0000 UTC m=+1949.433227795" watchObservedRunningTime="2026-07-29 19:33:03.134719196 +0000 UTC m=+1949.434391695"
Jul 29 19:33:03.156577 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:03.156525    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb" podStartSLOduration=1.156510895 podStartE2EDuration="1.156510895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:33:03.155886066 +0000 UTC m=+1949.455558567" watchObservedRunningTime="2026-07-29 19:33:03.156510895 +0000 UTC m=+1949.456183395"
Jul 29 19:33:09.103130 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:09.103101    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:14.100077 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:14.100038    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"
Jul 29 19:33:14.100594 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:14.100416    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"
Jul 29 19:33:41.757018 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:41.756982    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"]
Jul 29 19:33:41.757553 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:41.757179    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd" podUID="22f6c3c1-30dd-4ea1-93c2-682165982db1" containerName="maas-api" containerID="cri-o://55febe1ee5117467c5a584fb4e04a2bbd19d10d59496521be1d2f9eae3e4fea4" gracePeriod=30
Jul 29 19:33:41.811185 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:41.811145    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"]
Jul 29 19:33:41.811499 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:41.811444    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb" podUID="18fe946e-a1a6-42c8-bc32-77d2060d7288" containerName="payload-processing" containerID="cri-o://9170def1432ee01f03affd742b43035804b89381e81a14f2effcb69431e174a7" gracePeriod=30
Jul 29 19:33:41.818685 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:41.818657    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"]
Jul 29 19:33:41.818907 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:41.818869    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p" podUID="a9284f51-a1c5-4f11-a6bd-05cc9b0aff56" containerName="payload-pre-processing" containerID="cri-o://8327d990763c3aafd3333bd053ecc0bb73348c781debde8c4488ad0da2d1257a" gracePeriod=30
Jul 29 19:33:42.100299 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.100271    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"
Jul 29 19:33:42.123841 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.123814    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"
Jul 29 19:33:42.130754 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.130721    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:42.243544 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.243511    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl7gl\" (UniqueName: \"kubernetes.io/projected/22f6c3c1-30dd-4ea1-93c2-682165982db1-kube-api-access-pl7gl\") pod \"22f6c3c1-30dd-4ea1-93c2-682165982db1\" (UID: \"22f6c3c1-30dd-4ea1-93c2-682165982db1\") "
Jul 29 19:33:42.243719 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.243588    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22f6c3c1-30dd-4ea1-93c2-682165982db1-maas-api-tls\") pod \"22f6c3c1-30dd-4ea1-93c2-682165982db1\" (UID: \"22f6c3c1-30dd-4ea1-93c2-682165982db1\") "
Jul 29 19:33:42.243719 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.243610    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhnjw\" (UniqueName: \"kubernetes.io/projected/18fe946e-a1a6-42c8-bc32-77d2060d7288-kube-api-access-fhnjw\") pod \"18fe946e-a1a6-42c8-bc32-77d2060d7288\" (UID: \"18fe946e-a1a6-42c8-bc32-77d2060d7288\") "
Jul 29 19:33:42.243870 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.243777    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9284f51-a1c5-4f11-a6bd-05cc9b0aff56-plugins-config-volume\") pod \"a9284f51-a1c5-4f11-a6bd-05cc9b0aff56\" (UID: \"a9284f51-a1c5-4f11-a6bd-05cc9b0aff56\") "
Jul 29 19:33:42.243937 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.243900    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztqjr\" (UniqueName: \"kubernetes.io/projected/a9284f51-a1c5-4f11-a6bd-05cc9b0aff56-kube-api-access-ztqjr\") pod \"a9284f51-a1c5-4f11-a6bd-05cc9b0aff56\" (UID: \"a9284f51-a1c5-4f11-a6bd-05cc9b0aff56\") "
Jul 29 19:33:42.243937 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.243932    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f6c3c1-30dd-4ea1-93c2-682165982db1-openshift-service-ca\") pod \"22f6c3c1-30dd-4ea1-93c2-682165982db1\" (UID: \"22f6c3c1-30dd-4ea1-93c2-682165982db1\") "
Jul 29 19:33:42.244049 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.243986    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fe946e-a1a6-42c8-bc32-77d2060d7288-plugins-config-volume\") pod \"18fe946e-a1a6-42c8-bc32-77d2060d7288\" (UID: \"18fe946e-a1a6-42c8-bc32-77d2060d7288\") "
Jul 29 19:33:42.244233 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.244177    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9284f51-a1c5-4f11-a6bd-05cc9b0aff56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9284f51-a1c5-4f11-a6bd-05cc9b0aff56" (UID: "a9284f51-a1c5-4f11-a6bd-05cc9b0aff56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:33:42.244340 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.244327    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9284f51-a1c5-4f11-a6bd-05cc9b0aff56-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:33:42.244477 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.244448    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18fe946e-a1a6-42c8-bc32-77d2060d7288-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "18fe946e-a1a6-42c8-bc32-77d2060d7288" (UID: "18fe946e-a1a6-42c8-bc32-77d2060d7288"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:33:42.244811 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.244785    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f6c3c1-30dd-4ea1-93c2-682165982db1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "22f6c3c1-30dd-4ea1-93c2-682165982db1" (UID: "22f6c3c1-30dd-4ea1-93c2-682165982db1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:33:42.245907 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.245877    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f6c3c1-30dd-4ea1-93c2-682165982db1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "22f6c3c1-30dd-4ea1-93c2-682165982db1" (UID: "22f6c3c1-30dd-4ea1-93c2-682165982db1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:33:42.246359 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.246340    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f6c3c1-30dd-4ea1-93c2-682165982db1-kube-api-access-pl7gl" (OuterVolumeSpecName: "kube-api-access-pl7gl") pod "22f6c3c1-30dd-4ea1-93c2-682165982db1" (UID: "22f6c3c1-30dd-4ea1-93c2-682165982db1"). InnerVolumeSpecName "kube-api-access-pl7gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:33:42.246359 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.246349    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fe946e-a1a6-42c8-bc32-77d2060d7288-kube-api-access-fhnjw" (OuterVolumeSpecName: "kube-api-access-fhnjw") pod "18fe946e-a1a6-42c8-bc32-77d2060d7288" (UID: "18fe946e-a1a6-42c8-bc32-77d2060d7288"). InnerVolumeSpecName "kube-api-access-fhnjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:33:42.246487 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.246353    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9284f51-a1c5-4f11-a6bd-05cc9b0aff56-kube-api-access-ztqjr" (OuterVolumeSpecName: "kube-api-access-ztqjr") pod "a9284f51-a1c5-4f11-a6bd-05cc9b0aff56" (UID: "a9284f51-a1c5-4f11-a6bd-05cc9b0aff56"). InnerVolumeSpecName "kube-api-access-ztqjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:33:42.298552 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.298463    2579 generic.go:356] "Generic (PLEG): container finished" podID="22f6c3c1-30dd-4ea1-93c2-682165982db1" containerID="55febe1ee5117467c5a584fb4e04a2bbd19d10d59496521be1d2f9eae3e4fea4" exitCode=0
Jul 29 19:33:42.298552 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.298508    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"
Jul 29 19:33:42.298744 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.298553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd" event={"ID":"22f6c3c1-30dd-4ea1-93c2-682165982db1","Type":"ContainerDied","Data":"55febe1ee5117467c5a584fb4e04a2bbd19d10d59496521be1d2f9eae3e4fea4"}
Jul 29 19:33:42.298744 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.298603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd" event={"ID":"22f6c3c1-30dd-4ea1-93c2-682165982db1","Type":"ContainerDied","Data":"b083c99c3cc346ae3d11f279c86d905a3a76e80f3a8da159638b30a14994bedb"}
Jul 29 19:33:42.298744 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.298632    2579 scope.go:117] "RemoveContainer" containerID="55febe1ee5117467c5a584fb4e04a2bbd19d10d59496521be1d2f9eae3e4fea4"
Jul 29 19:33:42.299954 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.299917    2579 generic.go:356] "Generic (PLEG): container finished" podID="a9284f51-a1c5-4f11-a6bd-05cc9b0aff56" containerID="8327d990763c3aafd3333bd053ecc0bb73348c781debde8c4488ad0da2d1257a" exitCode=0
Jul 29 19:33:42.300078 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.299960    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"
Jul 29 19:33:42.300078 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.299967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p" event={"ID":"a9284f51-a1c5-4f11-a6bd-05cc9b0aff56","Type":"ContainerDied","Data":"8327d990763c3aafd3333bd053ecc0bb73348c781debde8c4488ad0da2d1257a"}
Jul 29 19:33:42.300078 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.300017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p" event={"ID":"a9284f51-a1c5-4f11-a6bd-05cc9b0aff56","Type":"ContainerDied","Data":"826d65dfe9359545af539b07ee58280bb67aa793437d071ed540d595f06e76bb"}
Jul 29 19:33:42.301535 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.301506    2579 generic.go:356] "Generic (PLEG): container finished" podID="18fe946e-a1a6-42c8-bc32-77d2060d7288" containerID="9170def1432ee01f03affd742b43035804b89381e81a14f2effcb69431e174a7" exitCode=0
Jul 29 19:33:42.301648 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.301533    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb" event={"ID":"18fe946e-a1a6-42c8-bc32-77d2060d7288","Type":"ContainerDied","Data":"9170def1432ee01f03affd742b43035804b89381e81a14f2effcb69431e174a7"}
Jul 29 19:33:42.301648 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.301557    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"
Jul 29 19:33:42.301648 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.301590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb" event={"ID":"18fe946e-a1a6-42c8-bc32-77d2060d7288","Type":"ContainerDied","Data":"9a58c6e2e35abb7568f243309574c6a3304e4c2638ef689f9f0f49c93e9ff518"}
Jul 29 19:33:42.309968 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.309948    2579 scope.go:117] "RemoveContainer" containerID="55febe1ee5117467c5a584fb4e04a2bbd19d10d59496521be1d2f9eae3e4fea4"
Jul 29 19:33:42.310442 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:33:42.310333    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55febe1ee5117467c5a584fb4e04a2bbd19d10d59496521be1d2f9eae3e4fea4\": container with ID starting with 55febe1ee5117467c5a584fb4e04a2bbd19d10d59496521be1d2f9eae3e4fea4 not found: ID does not exist" containerID="55febe1ee5117467c5a584fb4e04a2bbd19d10d59496521be1d2f9eae3e4fea4"
Jul 29 19:33:42.310442 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.310371    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55febe1ee5117467c5a584fb4e04a2bbd19d10d59496521be1d2f9eae3e4fea4"} err="failed to get container status \"55febe1ee5117467c5a584fb4e04a2bbd19d10d59496521be1d2f9eae3e4fea4\": rpc error: code = NotFound desc = could not find container \"55febe1ee5117467c5a584fb4e04a2bbd19d10d59496521be1d2f9eae3e4fea4\": container with ID starting with 55febe1ee5117467c5a584fb4e04a2bbd19d10d59496521be1d2f9eae3e4fea4 not found: ID does not exist"
Jul 29 19:33:42.310442 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.310396    2579 scope.go:117] "RemoveContainer" containerID="8327d990763c3aafd3333bd053ecc0bb73348c781debde8c4488ad0da2d1257a"
Jul 29 19:33:42.319027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.319006    2579 scope.go:117] "RemoveContainer" containerID="8327d990763c3aafd3333bd053ecc0bb73348c781debde8c4488ad0da2d1257a"
Jul 29 19:33:42.319332 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:33:42.319314    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8327d990763c3aafd3333bd053ecc0bb73348c781debde8c4488ad0da2d1257a\": container with ID starting with 8327d990763c3aafd3333bd053ecc0bb73348c781debde8c4488ad0da2d1257a not found: ID does not exist" containerID="8327d990763c3aafd3333bd053ecc0bb73348c781debde8c4488ad0da2d1257a"
Jul 29 19:33:42.319381 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.319340    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8327d990763c3aafd3333bd053ecc0bb73348c781debde8c4488ad0da2d1257a"} err="failed to get container status \"8327d990763c3aafd3333bd053ecc0bb73348c781debde8c4488ad0da2d1257a\": rpc error: code = NotFound desc = could not find container \"8327d990763c3aafd3333bd053ecc0bb73348c781debde8c4488ad0da2d1257a\": container with ID starting with 8327d990763c3aafd3333bd053ecc0bb73348c781debde8c4488ad0da2d1257a not found: ID does not exist"
Jul 29 19:33:42.319381 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.319358    2579 scope.go:117] "RemoveContainer" containerID="9170def1432ee01f03affd742b43035804b89381e81a14f2effcb69431e174a7"
Jul 29 19:33:42.327708 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.327681    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"]
Jul 29 19:33:42.328070 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.328046    2579 scope.go:117] "RemoveContainer" containerID="9170def1432ee01f03affd742b43035804b89381e81a14f2effcb69431e174a7"
Jul 29 19:33:42.328410 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:33:42.328386    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9170def1432ee01f03affd742b43035804b89381e81a14f2effcb69431e174a7\": container with ID starting with 9170def1432ee01f03affd742b43035804b89381e81a14f2effcb69431e174a7 not found: ID does not exist" containerID="9170def1432ee01f03affd742b43035804b89381e81a14f2effcb69431e174a7"
Jul 29 19:33:42.328491 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.328418    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9170def1432ee01f03affd742b43035804b89381e81a14f2effcb69431e174a7"} err="failed to get container status \"9170def1432ee01f03affd742b43035804b89381e81a14f2effcb69431e174a7\": rpc error: code = NotFound desc = could not find container \"9170def1432ee01f03affd742b43035804b89381e81a14f2effcb69431e174a7\": container with ID starting with 9170def1432ee01f03affd742b43035804b89381e81a14f2effcb69431e174a7 not found: ID does not exist"
Jul 29 19:33:42.331869 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.331842    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d42a5b6-6879cdfc97-9lvkd"]
Jul 29 19:33:42.345039 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.345004    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztqjr\" (UniqueName: \"kubernetes.io/projected/a9284f51-a1c5-4f11-a6bd-05cc9b0aff56-kube-api-access-ztqjr\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:33:42.345039 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.345034    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f6c3c1-30dd-4ea1-93c2-682165982db1-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:33:42.345039 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.345045    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fe946e-a1a6-42c8-bc32-77d2060d7288-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:33:42.345309 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.345056    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl7gl\" (UniqueName: \"kubernetes.io/projected/22f6c3c1-30dd-4ea1-93c2-682165982db1-kube-api-access-pl7gl\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:33:42.345309 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.345065    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22f6c3c1-30dd-4ea1-93c2-682165982db1-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:33:42.345309 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.345096    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhnjw\" (UniqueName: \"kubernetes.io/projected/18fe946e-a1a6-42c8-bc32-77d2060d7288-kube-api-access-fhnjw\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:33:42.351100 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.351071    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"]
Jul 29 19:33:42.356911 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.356880    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d42a5b6-5894db7964-4v72p"]
Jul 29 19:33:42.383471 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.383438    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"]
Jul 29 19:33:42.385024 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:42.385001    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d42a5b6-5668c74dbd-fqggb"]
Jul 29 19:33:44.202780 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:44.202741    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18fe946e-a1a6-42c8-bc32-77d2060d7288" path="/var/lib/kubelet/pods/18fe946e-a1a6-42c8-bc32-77d2060d7288/volumes"
Jul 29 19:33:44.203179 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:44.203145    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22f6c3c1-30dd-4ea1-93c2-682165982db1" path="/var/lib/kubelet/pods/22f6c3c1-30dd-4ea1-93c2-682165982db1/volumes"
Jul 29 19:33:44.203513 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:44.203498    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9284f51-a1c5-4f11-a6bd-05cc9b0aff56" path="/var/lib/kubelet/pods/a9284f51-a1c5-4f11-a6bd-05cc9b0aff56/volumes"
Jul 29 19:33:46.522215 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.522151    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"]
Jul 29 19:33:46.522740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.522564    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18fe946e-a1a6-42c8-bc32-77d2060d7288" containerName="payload-processing"
Jul 29 19:33:46.522740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.522577    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="18fe946e-a1a6-42c8-bc32-77d2060d7288" containerName="payload-processing"
Jul 29 19:33:46.522740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.522602    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f6c3c1-30dd-4ea1-93c2-682165982db1" containerName="maas-api"
Jul 29 19:33:46.522740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.522608    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f6c3c1-30dd-4ea1-93c2-682165982db1" containerName="maas-api"
Jul 29 19:33:46.522740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.522617    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9284f51-a1c5-4f11-a6bd-05cc9b0aff56" containerName="payload-pre-processing"
Jul 29 19:33:46.522740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.522623    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9284f51-a1c5-4f11-a6bd-05cc9b0aff56" containerName="payload-pre-processing"
Jul 29 19:33:46.522740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.522679    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9284f51-a1c5-4f11-a6bd-05cc9b0aff56" containerName="payload-pre-processing"
Jul 29 19:33:46.522740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.522687    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f6c3c1-30dd-4ea1-93c2-682165982db1" containerName="maas-api"
Jul 29 19:33:46.522740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.522696    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="18fe946e-a1a6-42c8-bc32-77d2060d7288" containerName="payload-processing"
Jul 29 19:33:46.527274 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.527254    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:33:46.531159 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.531137    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:33:46.532650 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.532628    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3a338fad-serving-cert\""
Jul 29 19:33:46.532778 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.532681    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:33:46.532778 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.532647    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5bzmf\""
Jul 29 19:33:46.537022 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.536997    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"]
Jul 29 19:33:46.687751 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.687703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbef8748-f332-4e3a-8616-f47be3e6d92b-maas-api-tls\") pod \"maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj\" (UID: \"dbef8748-f332-4e3a-8616-f47be3e6d92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:33:46.687981 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.687790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xms6l\" (UniqueName: \"kubernetes.io/projected/dbef8748-f332-4e3a-8616-f47be3e6d92b-kube-api-access-xms6l\") pod \"maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj\" (UID: \"dbef8748-f332-4e3a-8616-f47be3e6d92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:33:46.687981 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.687821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbef8748-f332-4e3a-8616-f47be3e6d92b-openshift-service-ca\") pod \"maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj\" (UID: \"dbef8748-f332-4e3a-8616-f47be3e6d92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:33:46.788723 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.788615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbef8748-f332-4e3a-8616-f47be3e6d92b-maas-api-tls\") pod \"maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj\" (UID: \"dbef8748-f332-4e3a-8616-f47be3e6d92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:33:46.788723 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.788680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xms6l\" (UniqueName: \"kubernetes.io/projected/dbef8748-f332-4e3a-8616-f47be3e6d92b-kube-api-access-xms6l\") pod \"maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj\" (UID: \"dbef8748-f332-4e3a-8616-f47be3e6d92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:33:46.788723 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.788705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbef8748-f332-4e3a-8616-f47be3e6d92b-openshift-service-ca\") pod \"maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj\" (UID: \"dbef8748-f332-4e3a-8616-f47be3e6d92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:33:46.789031 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:33:46.788768    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-serving-cert: secret "maas-api-e2e-derive-3a338fad-serving-cert" not found
Jul 29 19:33:46.789031 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:33:46.788840    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbef8748-f332-4e3a-8616-f47be3e6d92b-maas-api-tls podName:dbef8748-f332-4e3a-8616-f47be3e6d92b nodeName:}" failed. No retries permitted until 2026-07-29 19:33:47.288822292 +0000 UTC m=+1993.588494771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dbef8748-f332-4e3a-8616-f47be3e6d92b-maas-api-tls") pod "maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj" (UID: "dbef8748-f332-4e3a-8616-f47be3e6d92b") : secret "maas-api-e2e-derive-3a338fad-serving-cert" not found
Jul 29 19:33:46.789367 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.789349    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbef8748-f332-4e3a-8616-f47be3e6d92b-openshift-service-ca\") pod \"maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj\" (UID: \"dbef8748-f332-4e3a-8616-f47be3e6d92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:33:46.801416 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:46.801377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xms6l\" (UniqueName: \"kubernetes.io/projected/dbef8748-f332-4e3a-8616-f47be3e6d92b-kube-api-access-xms6l\") pod \"maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj\" (UID: \"dbef8748-f332-4e3a-8616-f47be3e6d92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:33:47.292394 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.292346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbef8748-f332-4e3a-8616-f47be3e6d92b-maas-api-tls\") pod \"maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj\" (UID: \"dbef8748-f332-4e3a-8616-f47be3e6d92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:33:47.295038 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.295002    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbef8748-f332-4e3a-8616-f47be3e6d92b-maas-api-tls\") pod \"maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj\" (UID: \"dbef8748-f332-4e3a-8616-f47be3e6d92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:33:47.362836 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.362797    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"]
Jul 29 19:33:47.366920 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.366894    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"
Jul 29 19:33:47.370361 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.370328    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3a338fad-dockercfg-cfjj5\""
Jul 29 19:33:47.370531 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.370483    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3a338fad\""
Jul 29 19:33:47.375913 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.375879    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"]
Jul 29 19:33:47.439509 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.439473    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:33:47.482210 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.482154    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"]
Jul 29 19:33:47.487364 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.487337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"
Jul 29 19:33:47.495930 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.495348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896fe73b-244f-4f26-9a4f-7cdbc47dfa7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t\" (UID: \"896fe73b-244f-4f26-9a4f-7cdbc47dfa7c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"
Jul 29 19:33:47.495930 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.495460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdw47\" (UniqueName: \"kubernetes.io/projected/896fe73b-244f-4f26-9a4f-7cdbc47dfa7c-kube-api-access-zdw47\") pod \"payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t\" (UID: \"896fe73b-244f-4f26-9a4f-7cdbc47dfa7c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"
Jul 29 19:33:47.508770 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.508717    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"]
Jul 29 19:33:47.596334 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.596250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdw47\" (UniqueName: \"kubernetes.io/projected/896fe73b-244f-4f26-9a4f-7cdbc47dfa7c-kube-api-access-zdw47\") pod \"payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t\" (UID: \"896fe73b-244f-4f26-9a4f-7cdbc47dfa7c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"
Jul 29 19:33:47.596334 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.596305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxmc8\" (UniqueName: \"kubernetes.io/projected/cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5-kube-api-access-dxmc8\") pod \"payload-processing-e2e-derive-3a338fad-6549496d4b-58njc\" (UID: \"cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5\") " pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"
Jul 29 19:33:47.596801 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.596340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5-plugins-config-volume\") pod \"payload-processing-e2e-derive-3a338fad-6549496d4b-58njc\" (UID: \"cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5\") " pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"
Jul 29 19:33:47.596801 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.596368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896fe73b-244f-4f26-9a4f-7cdbc47dfa7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t\" (UID: \"896fe73b-244f-4f26-9a4f-7cdbc47dfa7c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"
Jul 29 19:33:47.597000 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.596980    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896fe73b-244f-4f26-9a4f-7cdbc47dfa7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t\" (UID: \"896fe73b-244f-4f26-9a4f-7cdbc47dfa7c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"
Jul 29 19:33:47.614615 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.614577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdw47\" (UniqueName: \"kubernetes.io/projected/896fe73b-244f-4f26-9a4f-7cdbc47dfa7c-kube-api-access-zdw47\") pod \"payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t\" (UID: \"896fe73b-244f-4f26-9a4f-7cdbc47dfa7c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"
Jul 29 19:33:47.636795 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.636764    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"]
Jul 29 19:33:47.638835 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:33:47.638794    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbef8748_f332_4e3a_8616_f47be3e6d92b.slice/crio-86367677d5eb9c98f3dd3c207accd68bf3e0982ef4c64ef8985dad8679a5d852 WatchSource:0}: Error finding container 86367677d5eb9c98f3dd3c207accd68bf3e0982ef4c64ef8985dad8679a5d852: Status 404 returned error can't find the container with id 86367677d5eb9c98f3dd3c207accd68bf3e0982ef4c64ef8985dad8679a5d852
Jul 29 19:33:47.680045 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.680011    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"
Jul 29 19:33:47.697889 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.697859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxmc8\" (UniqueName: \"kubernetes.io/projected/cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5-kube-api-access-dxmc8\") pod \"payload-processing-e2e-derive-3a338fad-6549496d4b-58njc\" (UID: \"cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5\") " pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"
Jul 29 19:33:47.698045 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.697908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5-plugins-config-volume\") pod \"payload-processing-e2e-derive-3a338fad-6549496d4b-58njc\" (UID: \"cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5\") " pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"
Jul 29 19:33:47.698600 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.698582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5-plugins-config-volume\") pod \"payload-processing-e2e-derive-3a338fad-6549496d4b-58njc\" (UID: \"cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5\") " pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"
Jul 29 19:33:47.721472 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.721444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxmc8\" (UniqueName: \"kubernetes.io/projected/cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5-kube-api-access-dxmc8\") pod \"payload-processing-e2e-derive-3a338fad-6549496d4b-58njc\" (UID: \"cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5\") " pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"
Jul 29 19:33:47.800975 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.800931    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"
Jul 29 19:33:47.845726 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.845694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"]
Jul 29 19:33:47.848549 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:33:47.848469    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896fe73b_244f_4f26_9a4f_7cdbc47dfa7c.slice/crio-5bf05b89492b645d598cf645ff6312d11fea21ea4206c01c5e839cfa055d2bc2 WatchSource:0}: Error finding container 5bf05b89492b645d598cf645ff6312d11fea21ea4206c01c5e839cfa055d2bc2: Status 404 returned error can't find the container with id 5bf05b89492b645d598cf645ff6312d11fea21ea4206c01c5e839cfa055d2bc2
Jul 29 19:33:47.967482 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:47.967444    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"]
Jul 29 19:33:47.970769 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:33:47.970738    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbac65c7_1c79_4d39_b1f6_f44f70c7cdc5.slice/crio-1e4b45bed036ef86b4fb94f2544ea949b5dd730785ddddda562277b27661d2a0 WatchSource:0}: Error finding container 1e4b45bed036ef86b4fb94f2544ea949b5dd730785ddddda562277b27661d2a0: Status 404 returned error can't find the container with id 1e4b45bed036ef86b4fb94f2544ea949b5dd730785ddddda562277b27661d2a0
Jul 29 19:33:48.337005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:48.336964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc" event={"ID":"cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5","Type":"ContainerStarted","Data":"797f0ba5dff575179b25b8dafdc203a2b3bfc2334ff399175d0d19822a96059e"}
Jul 29 19:33:48.337232 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:48.337018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc" event={"ID":"cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5","Type":"ContainerStarted","Data":"1e4b45bed036ef86b4fb94f2544ea949b5dd730785ddddda562277b27661d2a0"}
Jul 29 19:33:48.337232 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:48.337074    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"
Jul 29 19:33:48.338675 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:48.338644    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj" event={"ID":"dbef8748-f332-4e3a-8616-f47be3e6d92b","Type":"ContainerStarted","Data":"f16a66b8cec2412b4a114aba553eaa64b8dc6344dd5288b40f6ca05681f38ff7"}
Jul 29 19:33:48.338823 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:48.338696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj" event={"ID":"dbef8748-f332-4e3a-8616-f47be3e6d92b","Type":"ContainerStarted","Data":"86367677d5eb9c98f3dd3c207accd68bf3e0982ef4c64ef8985dad8679a5d852"}
Jul 29 19:33:48.338823 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:48.338723    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:33:48.340203 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:48.340161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t" event={"ID":"896fe73b-244f-4f26-9a4f-7cdbc47dfa7c","Type":"ContainerStarted","Data":"465238f3c1cf451192878347e4657407c74851bf90efd02ff8bf7e41574ad6fc"}
Jul 29 19:33:48.340323 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:48.340223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t" event={"ID":"896fe73b-244f-4f26-9a4f-7cdbc47dfa7c","Type":"ContainerStarted","Data":"5bf05b89492b645d598cf645ff6312d11fea21ea4206c01c5e839cfa055d2bc2"}
Jul 29 19:33:48.340323 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:48.340252    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"
Jul 29 19:33:48.354917 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:48.354861    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc" podStartSLOduration=1.3548455289999999 podStartE2EDuration="1.354845529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:33:48.353955357 +0000 UTC m=+1994.653627858" watchObservedRunningTime="2026-07-29 19:33:48.354845529 +0000 UTC m=+1994.654518030"
Jul 29 19:33:48.372646 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:48.372526    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t" podStartSLOduration=1.372510992 podStartE2EDuration="1.372510992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:33:48.371318246 +0000 UTC m=+1994.670990748" watchObservedRunningTime="2026-07-29 19:33:48.372510992 +0000 UTC m=+1994.672183492"
Jul 29 19:33:54.351788 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:54.351755    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:33:54.372116 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:54.372061    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj" podStartSLOduration=7.958945402 podStartE2EDuration="8.372044975s" totalImagesPullingTime="413.099573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:33:48.391164692 +0000 UTC m=+1994.690837192" watchObservedRunningTime="2026-07-29 19:33:54.372044975 +0000 UTC m=+2000.671717476"
Jul 29 19:33:59.347956 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:59.347926    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"
Jul 29 19:33:59.348397 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:33:59.347996    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"
Jul 29 19:34:17.031176 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.031144    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"]
Jul 29 19:34:17.031642 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.031390    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj" podUID="dbef8748-f332-4e3a-8616-f47be3e6d92b" containerName="maas-api" containerID="cri-o://f16a66b8cec2412b4a114aba553eaa64b8dc6344dd5288b40f6ca05681f38ff7" gracePeriod=30
Jul 29 19:34:17.152084 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.152051    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"]
Jul 29 19:34:17.152321 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.152276    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc" podUID="cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5" containerName="payload-processing" containerID="cri-o://797f0ba5dff575179b25b8dafdc203a2b3bfc2334ff399175d0d19822a96059e" gracePeriod=30
Jul 29 19:34:17.167381 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.167347    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"]
Jul 29 19:34:17.167780 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.167607    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t" podUID="896fe73b-244f-4f26-9a4f-7cdbc47dfa7c" containerName="payload-pre-processing" containerID="cri-o://465238f3c1cf451192878347e4657407c74851bf90efd02ff8bf7e41574ad6fc" gracePeriod=30
Jul 29 19:34:17.290596 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.290568    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:34:17.363724 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.363692    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xms6l\" (UniqueName: \"kubernetes.io/projected/dbef8748-f332-4e3a-8616-f47be3e6d92b-kube-api-access-xms6l\") pod \"dbef8748-f332-4e3a-8616-f47be3e6d92b\" (UID: \"dbef8748-f332-4e3a-8616-f47be3e6d92b\") "
Jul 29 19:34:17.363880 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.363745    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbef8748-f332-4e3a-8616-f47be3e6d92b-openshift-service-ca\") pod \"dbef8748-f332-4e3a-8616-f47be3e6d92b\" (UID: \"dbef8748-f332-4e3a-8616-f47be3e6d92b\") "
Jul 29 19:34:17.363880 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.363788    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbef8748-f332-4e3a-8616-f47be3e6d92b-maas-api-tls\") pod \"dbef8748-f332-4e3a-8616-f47be3e6d92b\" (UID: \"dbef8748-f332-4e3a-8616-f47be3e6d92b\") "
Jul 29 19:34:17.364157 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.364133    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbef8748-f332-4e3a-8616-f47be3e6d92b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dbef8748-f332-4e3a-8616-f47be3e6d92b" (UID: "dbef8748-f332-4e3a-8616-f47be3e6d92b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:34:17.367445 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.367412    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbef8748-f332-4e3a-8616-f47be3e6d92b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dbef8748-f332-4e3a-8616-f47be3e6d92b" (UID: "dbef8748-f332-4e3a-8616-f47be3e6d92b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:34:17.367551 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.367462    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbef8748-f332-4e3a-8616-f47be3e6d92b-kube-api-access-xms6l" (OuterVolumeSpecName: "kube-api-access-xms6l") pod "dbef8748-f332-4e3a-8616-f47be3e6d92b" (UID: "dbef8748-f332-4e3a-8616-f47be3e6d92b"). InnerVolumeSpecName "kube-api-access-xms6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:34:17.428157 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.428125    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"
Jul 29 19:34:17.435831 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.435810    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"
Jul 29 19:34:17.464547 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.464522    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5-plugins-config-volume\") pod \"cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5\" (UID: \"cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5\") "
Jul 29 19:34:17.464698 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.464634    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxmc8\" (UniqueName: \"kubernetes.io/projected/cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5-kube-api-access-dxmc8\") pod \"cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5\" (UID: \"cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5\") "
Jul 29 19:34:17.464894 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.464873    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xms6l\" (UniqueName: \"kubernetes.io/projected/dbef8748-f332-4e3a-8616-f47be3e6d92b-kube-api-access-xms6l\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:34:17.464963 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.464895    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbef8748-f332-4e3a-8616-f47be3e6d92b-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:34:17.464963 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.464909    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbef8748-f332-4e3a-8616-f47be3e6d92b-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:34:17.464963 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.464919    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5" (UID: "cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:34:17.466927 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.466908    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5-kube-api-access-dxmc8" (OuterVolumeSpecName: "kube-api-access-dxmc8") pod "cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5" (UID: "cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5"). InnerVolumeSpecName "kube-api-access-dxmc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:34:17.496555 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.496224    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"
Jul 29 19:34:17.497399 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.497176    2579 generic.go:356] "Generic (PLEG): container finished" podID="cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5" containerID="797f0ba5dff575179b25b8dafdc203a2b3bfc2334ff399175d0d19822a96059e" exitCode=0
Jul 29 19:34:17.497399 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.497319    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc" event={"ID":"cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5","Type":"ContainerDied","Data":"797f0ba5dff575179b25b8dafdc203a2b3bfc2334ff399175d0d19822a96059e"}
Jul 29 19:34:17.497399 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.497366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc" event={"ID":"cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5","Type":"ContainerDied","Data":"1e4b45bed036ef86b4fb94f2544ea949b5dd730785ddddda562277b27661d2a0"}
Jul 29 19:34:17.497399 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.497396    2579 scope.go:117] "RemoveContainer" containerID="797f0ba5dff575179b25b8dafdc203a2b3bfc2334ff399175d0d19822a96059e"
Jul 29 19:34:17.499480 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.499445    2579 generic.go:356] "Generic (PLEG): container finished" podID="dbef8748-f332-4e3a-8616-f47be3e6d92b" containerID="f16a66b8cec2412b4a114aba553eaa64b8dc6344dd5288b40f6ca05681f38ff7" exitCode=0
Jul 29 19:34:17.499607 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.499589    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj" event={"ID":"dbef8748-f332-4e3a-8616-f47be3e6d92b","Type":"ContainerDied","Data":"f16a66b8cec2412b4a114aba553eaa64b8dc6344dd5288b40f6ca05681f38ff7"}
Jul 29 19:34:17.499687 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.499627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj" event={"ID":"dbef8748-f332-4e3a-8616-f47be3e6d92b","Type":"ContainerDied","Data":"86367677d5eb9c98f3dd3c207accd68bf3e0982ef4c64ef8985dad8679a5d852"}
Jul 29 19:34:17.499748 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.499687    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"
Jul 29 19:34:17.501667 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.501388    2579 generic.go:356] "Generic (PLEG): container finished" podID="896fe73b-244f-4f26-9a4f-7cdbc47dfa7c" containerID="465238f3c1cf451192878347e4657407c74851bf90efd02ff8bf7e41574ad6fc" exitCode=0
Jul 29 19:34:17.501667 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.501433    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t" event={"ID":"896fe73b-244f-4f26-9a4f-7cdbc47dfa7c","Type":"ContainerDied","Data":"465238f3c1cf451192878347e4657407c74851bf90efd02ff8bf7e41574ad6fc"}
Jul 29 19:34:17.501667 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.501494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t" event={"ID":"896fe73b-244f-4f26-9a4f-7cdbc47dfa7c","Type":"ContainerDied","Data":"5bf05b89492b645d598cf645ff6312d11fea21ea4206c01c5e839cfa055d2bc2"}
Jul 29 19:34:17.501667 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.501559    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"
Jul 29 19:34:17.510383 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.510362    2579 scope.go:117] "RemoveContainer" containerID="797f0ba5dff575179b25b8dafdc203a2b3bfc2334ff399175d0d19822a96059e"
Jul 29 19:34:17.510694 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:34:17.510672    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797f0ba5dff575179b25b8dafdc203a2b3bfc2334ff399175d0d19822a96059e\": container with ID starting with 797f0ba5dff575179b25b8dafdc203a2b3bfc2334ff399175d0d19822a96059e not found: ID does not exist" containerID="797f0ba5dff575179b25b8dafdc203a2b3bfc2334ff399175d0d19822a96059e"
Jul 29 19:34:17.510784 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.510714    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797f0ba5dff575179b25b8dafdc203a2b3bfc2334ff399175d0d19822a96059e"} err="failed to get container status \"797f0ba5dff575179b25b8dafdc203a2b3bfc2334ff399175d0d19822a96059e\": rpc error: code = NotFound desc = could not find container \"797f0ba5dff575179b25b8dafdc203a2b3bfc2334ff399175d0d19822a96059e\": container with ID starting with 797f0ba5dff575179b25b8dafdc203a2b3bfc2334ff399175d0d19822a96059e not found: ID does not exist"
Jul 29 19:34:17.510784 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.510730    2579 scope.go:117] "RemoveContainer" containerID="f16a66b8cec2412b4a114aba553eaa64b8dc6344dd5288b40f6ca05681f38ff7"
Jul 29 19:34:17.519548 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.519532    2579 scope.go:117] "RemoveContainer" containerID="f16a66b8cec2412b4a114aba553eaa64b8dc6344dd5288b40f6ca05681f38ff7"
Jul 29 19:34:17.519810 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:34:17.519794    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f16a66b8cec2412b4a114aba553eaa64b8dc6344dd5288b40f6ca05681f38ff7\": container with ID starting with f16a66b8cec2412b4a114aba553eaa64b8dc6344dd5288b40f6ca05681f38ff7 not found: ID does not exist" containerID="f16a66b8cec2412b4a114aba553eaa64b8dc6344dd5288b40f6ca05681f38ff7"
Jul 29 19:34:17.519857 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.519817    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f16a66b8cec2412b4a114aba553eaa64b8dc6344dd5288b40f6ca05681f38ff7"} err="failed to get container status \"f16a66b8cec2412b4a114aba553eaa64b8dc6344dd5288b40f6ca05681f38ff7\": rpc error: code = NotFound desc = could not find container \"f16a66b8cec2412b4a114aba553eaa64b8dc6344dd5288b40f6ca05681f38ff7\": container with ID starting with f16a66b8cec2412b4a114aba553eaa64b8dc6344dd5288b40f6ca05681f38ff7 not found: ID does not exist"
Jul 29 19:34:17.519857 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.519833    2579 scope.go:117] "RemoveContainer" containerID="465238f3c1cf451192878347e4657407c74851bf90efd02ff8bf7e41574ad6fc"
Jul 29 19:34:17.527326 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.527293    2579 scope.go:117] "RemoveContainer" containerID="465238f3c1cf451192878347e4657407c74851bf90efd02ff8bf7e41574ad6fc"
Jul 29 19:34:17.527652 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:34:17.527634    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465238f3c1cf451192878347e4657407c74851bf90efd02ff8bf7e41574ad6fc\": container with ID starting with 465238f3c1cf451192878347e4657407c74851bf90efd02ff8bf7e41574ad6fc not found: ID does not exist" containerID="465238f3c1cf451192878347e4657407c74851bf90efd02ff8bf7e41574ad6fc"
Jul 29 19:34:17.527719 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.527658    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465238f3c1cf451192878347e4657407c74851bf90efd02ff8bf7e41574ad6fc"} err="failed to get container status \"465238f3c1cf451192878347e4657407c74851bf90efd02ff8bf7e41574ad6fc\": rpc error: code = NotFound desc = could not find container \"465238f3c1cf451192878347e4657407c74851bf90efd02ff8bf7e41574ad6fc\": container with ID starting with 465238f3c1cf451192878347e4657407c74851bf90efd02ff8bf7e41574ad6fc not found: ID does not exist"
Jul 29 19:34:17.541526 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.541479    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"]
Jul 29 19:34:17.549113 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.549093    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3a338fad-6ff9699677-lp5hj"]
Jul 29 19:34:17.565448 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.565429    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896fe73b-244f-4f26-9a4f-7cdbc47dfa7c-plugins-config-volume\") pod \"896fe73b-244f-4f26-9a4f-7cdbc47dfa7c\" (UID: \"896fe73b-244f-4f26-9a4f-7cdbc47dfa7c\") "
Jul 29 19:34:17.565563 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.565461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdw47\" (UniqueName: \"kubernetes.io/projected/896fe73b-244f-4f26-9a4f-7cdbc47dfa7c-kube-api-access-zdw47\") pod \"896fe73b-244f-4f26-9a4f-7cdbc47dfa7c\" (UID: \"896fe73b-244f-4f26-9a4f-7cdbc47dfa7c\") "
Jul 29 19:34:17.565741 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.565719    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxmc8\" (UniqueName: \"kubernetes.io/projected/cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5-kube-api-access-dxmc8\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:34:17.565741 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.565742    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:34:17.565904 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.565830    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896fe73b-244f-4f26-9a4f-7cdbc47dfa7c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "896fe73b-244f-4f26-9a4f-7cdbc47dfa7c" (UID: "896fe73b-244f-4f26-9a4f-7cdbc47dfa7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:34:17.567631 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.567610    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896fe73b-244f-4f26-9a4f-7cdbc47dfa7c-kube-api-access-zdw47" (OuterVolumeSpecName: "kube-api-access-zdw47") pod "896fe73b-244f-4f26-9a4f-7cdbc47dfa7c" (UID: "896fe73b-244f-4f26-9a4f-7cdbc47dfa7c"). InnerVolumeSpecName "kube-api-access-zdw47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:34:17.583990 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.583966    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"]
Jul 29 19:34:17.587397 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.587373    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3a338fad-6549496d4b-58njc"]
Jul 29 19:34:17.666466 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.666433    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896fe73b-244f-4f26-9a4f-7cdbc47dfa7c-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:34:17.666466 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.666461    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdw47\" (UniqueName: \"kubernetes.io/projected/896fe73b-244f-4f26-9a4f-7cdbc47dfa7c-kube-api-access-zdw47\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:34:17.825372 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.825341    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"]
Jul 29 19:34:17.831672 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:17.831650    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3a338fad-658cf6ffb-fkm8t"]
Jul 29 19:34:18.197339 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:18.197299    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="896fe73b-244f-4f26-9a4f-7cdbc47dfa7c" path="/var/lib/kubelet/pods/896fe73b-244f-4f26-9a4f-7cdbc47dfa7c/volumes"
Jul 29 19:34:18.197784 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:18.197697    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5" path="/var/lib/kubelet/pods/cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5/volumes"
Jul 29 19:34:18.198057 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:34:18.198042    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbef8748-f332-4e3a-8616-f47be3e6d92b" path="/var/lib/kubelet/pods/dbef8748-f332-4e3a-8616-f47be3e6d92b/volumes"
Jul 29 19:35:34.268402 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:35:34.268372    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:35:34.280351 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:35:34.280329    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:36:09.986417 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.986389    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"]
Jul 29 19:36:09.986879 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.986816    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5" containerName="payload-processing"
Jul 29 19:36:09.986879 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.986834    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5" containerName="payload-processing"
Jul 29 19:36:09.986879 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.986867    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbef8748-f332-4e3a-8616-f47be3e6d92b" containerName="maas-api"
Jul 29 19:36:09.986879 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.986877    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbef8748-f332-4e3a-8616-f47be3e6d92b" containerName="maas-api"
Jul 29 19:36:09.987011 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.986915    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="896fe73b-244f-4f26-9a4f-7cdbc47dfa7c" containerName="payload-pre-processing"
Jul 29 19:36:09.987011 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.986923    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="896fe73b-244f-4f26-9a4f-7cdbc47dfa7c" containerName="payload-pre-processing"
Jul 29 19:36:09.987078 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.987031    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="896fe73b-244f-4f26-9a4f-7cdbc47dfa7c" containerName="payload-pre-processing"
Jul 29 19:36:09.987078 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.987042    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbef8748-f332-4e3a-8616-f47be3e6d92b" containerName="maas-api"
Jul 29 19:36:09.987078 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.987053    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbac65c7-1c79-4d39-b1f6-f44f70c7cdc5" containerName="payload-processing"
Jul 29 19:36:09.990072 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.990056    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:09.994006 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.993985    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7beba8e5-serving-cert\""
Jul 29 19:36:09.994113 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.994025    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:36:09.994720 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.994707    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5bzmf\""
Jul 29 19:36:09.995293 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:09.995269    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:36:10.006711 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.006688    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"]
Jul 29 19:36:10.145376 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.145343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj5sp\" (UniqueName: \"kubernetes.io/projected/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-kube-api-access-gj5sp\") pod \"maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52\" (UID: \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:10.145511 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.145421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-openshift-service-ca\") pod \"maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52\" (UID: \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:10.145511 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.145481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-maas-api-tls\") pod \"maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52\" (UID: \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:10.246425 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.246358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj5sp\" (UniqueName: \"kubernetes.io/projected/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-kube-api-access-gj5sp\") pod \"maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52\" (UID: \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:10.246425 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.246396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-openshift-service-ca\") pod \"maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52\" (UID: \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:10.246601 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.246447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-maas-api-tls\") pod \"maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52\" (UID: \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:10.247184 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.247128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-openshift-service-ca\") pod \"maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52\" (UID: \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:10.249003 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.248984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-maas-api-tls\") pod \"maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52\" (UID: \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:10.268569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.268544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj5sp\" (UniqueName: \"kubernetes.io/projected/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-kube-api-access-gj5sp\") pod \"maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52\" (UID: \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:10.301572 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.301545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:10.450042 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.450014    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"]
Jul 29 19:36:10.450375 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:36:10.450347    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed8ac6d_17da_4948_b587_9a41bda5e0e8.slice/crio-8230a8c267b06e11821fa90680042a9e241cd311322137a73c2a9e35345166e4 WatchSource:0}: Error finding container 8230a8c267b06e11821fa90680042a9e241cd311322137a73c2a9e35345166e4: Status 404 returned error can't find the container with id 8230a8c267b06e11821fa90680042a9e241cd311322137a73c2a9e35345166e4
Jul 29 19:36:10.953348 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.953325    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"]
Jul 29 19:36:10.957759 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.957741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"
Jul 29 19:36:10.961086 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.961064    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7beba8e5\""
Jul 29 19:36:10.962272 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.962254    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7beba8e5-dockercfg-gt96f\""
Jul 29 19:36:10.972293 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:10.972273    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"]
Jul 29 19:36:11.004381 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.004358    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"]
Jul 29 19:36:11.007575 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.007559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"
Jul 29 19:36:11.028885 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.028860    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"]
Jul 29 19:36:11.055255 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.055225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f723a8-b5fd-4d98-9e48-998873d1acda-plugins-config-volume\") pod \"payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq\" (UID: \"c6f723a8-b5fd-4d98-9e48-998873d1acda\") " pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"
Jul 29 19:36:11.055382 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.055271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m978k\" (UniqueName: \"kubernetes.io/projected/c6f723a8-b5fd-4d98-9e48-998873d1acda-kube-api-access-m978k\") pod \"payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq\" (UID: \"c6f723a8-b5fd-4d98-9e48-998873d1acda\") " pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"
Jul 29 19:36:11.057519 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.057493    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52" event={"ID":"3ed8ac6d-17da-4948-b587-9a41bda5e0e8","Type":"ContainerStarted","Data":"b2c1a947ecfb31f3bb97664629731defc79f29636b4d1cf53640b1a0a6ef58ef"}
Jul 29 19:36:11.057610 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.057540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52" event={"ID":"3ed8ac6d-17da-4948-b587-9a41bda5e0e8","Type":"ContainerStarted","Data":"8230a8c267b06e11821fa90680042a9e241cd311322137a73c2a9e35345166e4"}
Jul 29 19:36:11.057610 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.057592    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:11.077505 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.077465    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52" podStartSLOduration=1.618384239 podStartE2EDuration="2.077451522s" totalImagesPullingTime="459.067283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:36:11.07657262 +0000 UTC m=+2137.376245120" watchObservedRunningTime="2026-07-29 19:36:11.077451522 +0000 UTC m=+2137.377124021"
Jul 29 19:36:11.156249 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.156223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42650d1f-4de2-4db8-810d-f9b9296e1962-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w\" (UID: \"42650d1f-4de2-4db8-810d-f9b9296e1962\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"
Jul 29 19:36:11.156357 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.156342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hftqt\" (UniqueName: \"kubernetes.io/projected/42650d1f-4de2-4db8-810d-f9b9296e1962-kube-api-access-hftqt\") pod \"payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w\" (UID: \"42650d1f-4de2-4db8-810d-f9b9296e1962\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"
Jul 29 19:36:11.156444 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.156427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f723a8-b5fd-4d98-9e48-998873d1acda-plugins-config-volume\") pod \"payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq\" (UID: \"c6f723a8-b5fd-4d98-9e48-998873d1acda\") " pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"
Jul 29 19:36:11.156501 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.156484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m978k\" (UniqueName: \"kubernetes.io/projected/c6f723a8-b5fd-4d98-9e48-998873d1acda-kube-api-access-m978k\") pod \"payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq\" (UID: \"c6f723a8-b5fd-4d98-9e48-998873d1acda\") " pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"
Jul 29 19:36:11.157027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.157008    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f723a8-b5fd-4d98-9e48-998873d1acda-plugins-config-volume\") pod \"payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq\" (UID: \"c6f723a8-b5fd-4d98-9e48-998873d1acda\") " pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"
Jul 29 19:36:11.165187 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.165135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m978k\" (UniqueName: \"kubernetes.io/projected/c6f723a8-b5fd-4d98-9e48-998873d1acda-kube-api-access-m978k\") pod \"payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq\" (UID: \"c6f723a8-b5fd-4d98-9e48-998873d1acda\") " pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"
Jul 29 19:36:11.257497 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.257476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42650d1f-4de2-4db8-810d-f9b9296e1962-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w\" (UID: \"42650d1f-4de2-4db8-810d-f9b9296e1962\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"
Jul 29 19:36:11.257603 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.257507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hftqt\" (UniqueName: \"kubernetes.io/projected/42650d1f-4de2-4db8-810d-f9b9296e1962-kube-api-access-hftqt\") pod \"payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w\" (UID: \"42650d1f-4de2-4db8-810d-f9b9296e1962\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"
Jul 29 19:36:11.258111 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.258091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42650d1f-4de2-4db8-810d-f9b9296e1962-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w\" (UID: \"42650d1f-4de2-4db8-810d-f9b9296e1962\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"
Jul 29 19:36:11.266572 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.266549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hftqt\" (UniqueName: \"kubernetes.io/projected/42650d1f-4de2-4db8-810d-f9b9296e1962-kube-api-access-hftqt\") pod \"payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w\" (UID: \"42650d1f-4de2-4db8-810d-f9b9296e1962\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"
Jul 29 19:36:11.269383 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.269363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"
Jul 29 19:36:11.316671 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.316643    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"
Jul 29 19:36:11.468210 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.468163    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"]
Jul 29 19:36:11.469867 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:36:11.469835    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42650d1f_4de2_4db8_810d_f9b9296e1962.slice/crio-afc9512408b1c53b5a87882bf8522368986e69e86ebec418ed9c2f90096fc9c3 WatchSource:0}: Error finding container afc9512408b1c53b5a87882bf8522368986e69e86ebec418ed9c2f90096fc9c3: Status 404 returned error can't find the container with id afc9512408b1c53b5a87882bf8522368986e69e86ebec418ed9c2f90096fc9c3
Jul 29 19:36:11.633510 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:11.633478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"]
Jul 29 19:36:12.064518 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:12.064416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq" event={"ID":"c6f723a8-b5fd-4d98-9e48-998873d1acda","Type":"ContainerStarted","Data":"f9a8e0b1ce7e7577416cb324412938d144486c2c674674302220fee4ff0d5470"}
Jul 29 19:36:12.064518 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:12.064486    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"
Jul 29 19:36:12.064518 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:12.064512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq" event={"ID":"c6f723a8-b5fd-4d98-9e48-998873d1acda","Type":"ContainerStarted","Data":"b873f39cf2834407eb29116f7ae172ce50943412ab945c8e8f006b5844775b02"}
Jul 29 19:36:12.065933 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:12.065906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w" event={"ID":"42650d1f-4de2-4db8-810d-f9b9296e1962","Type":"ContainerStarted","Data":"94d7f47a18916b3b124e8e9de47dcf09959226126120258114596ae8401e0e2c"}
Jul 29 19:36:12.066040 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:12.065952    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w" event={"ID":"42650d1f-4de2-4db8-810d-f9b9296e1962","Type":"ContainerStarted","Data":"afc9512408b1c53b5a87882bf8522368986e69e86ebec418ed9c2f90096fc9c3"}
Jul 29 19:36:12.066083 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:12.066056    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"
Jul 29 19:36:12.088737 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:12.088696    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq" podStartSLOduration=2.088683993 podStartE2EDuration="2.088683993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:36:12.086848055 +0000 UTC m=+2138.386520554" watchObservedRunningTime="2026-07-29 19:36:12.088683993 +0000 UTC m=+2138.388356492"
Jul 29 19:36:12.111482 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:12.111444    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w" podStartSLOduration=2.111432534 podStartE2EDuration="2.111432534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:36:12.11022843 +0000 UTC m=+2138.409900928" watchObservedRunningTime="2026-07-29 19:36:12.111432534 +0000 UTC m=+2138.411105033"
Jul 29 19:36:17.070731 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:17.070705    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:23.073023 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:23.072991    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"
Jul 29 19:36:23.073482 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:23.073073    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"
Jul 29 19:36:52.840735 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:52.840645    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"]
Jul 29 19:36:52.841338 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:52.840845    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52" podUID="3ed8ac6d-17da-4948-b587-9a41bda5e0e8" containerName="maas-api" containerID="cri-o://b2c1a947ecfb31f3bb97664629731defc79f29636b4d1cf53640b1a0a6ef58ef" gracePeriod=30
Jul 29 19:36:52.894537 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:52.894487    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"]
Jul 29 19:36:52.894760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:52.894731    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq" podUID="c6f723a8-b5fd-4d98-9e48-998873d1acda" containerName="payload-processing" containerID="cri-o://f9a8e0b1ce7e7577416cb324412938d144486c2c674674302220fee4ff0d5470" gracePeriod=30
Jul 29 19:36:52.908547 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:52.908522    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"]
Jul 29 19:36:52.908763 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:52.908737    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w" podUID="42650d1f-4de2-4db8-810d-f9b9296e1962" containerName="payload-pre-processing" containerID="cri-o://94d7f47a18916b3b124e8e9de47dcf09959226126120258114596ae8401e0e2c" gracePeriod=30
Jul 29 19:36:53.072893 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.072581    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w" podUID="42650d1f-4de2-4db8-810d-f9b9296e1962" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.58:9004: connect: connection refused"
Jul 29 19:36:53.161160 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.161135    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"
Jul 29 19:36:53.187508 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.187485    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"
Jul 29 19:36:53.193581 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.193565    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:53.270718 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.270691    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ed8ac6d-17da-4948-b587-9a41bda5e0e8" containerID="b2c1a947ecfb31f3bb97664629731defc79f29636b4d1cf53640b1a0a6ef58ef" exitCode=0
Jul 29 19:36:53.270886 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.270732    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"
Jul 29 19:36:53.270886 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.270778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52" event={"ID":"3ed8ac6d-17da-4948-b587-9a41bda5e0e8","Type":"ContainerDied","Data":"b2c1a947ecfb31f3bb97664629731defc79f29636b4d1cf53640b1a0a6ef58ef"}
Jul 29 19:36:53.270886 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.270829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52" event={"ID":"3ed8ac6d-17da-4948-b587-9a41bda5e0e8","Type":"ContainerDied","Data":"8230a8c267b06e11821fa90680042a9e241cd311322137a73c2a9e35345166e4"}
Jul 29 19:36:53.270886 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.270850    2579 scope.go:117] "RemoveContainer" containerID="b2c1a947ecfb31f3bb97664629731defc79f29636b4d1cf53640b1a0a6ef58ef"
Jul 29 19:36:53.271968 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.271948    2579 generic.go:356] "Generic (PLEG): container finished" podID="c6f723a8-b5fd-4d98-9e48-998873d1acda" containerID="f9a8e0b1ce7e7577416cb324412938d144486c2c674674302220fee4ff0d5470" exitCode=0
Jul 29 19:36:53.272065 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.271985    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"
Jul 29 19:36:53.272065 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.272019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq" event={"ID":"c6f723a8-b5fd-4d98-9e48-998873d1acda","Type":"ContainerDied","Data":"f9a8e0b1ce7e7577416cb324412938d144486c2c674674302220fee4ff0d5470"}
Jul 29 19:36:53.272065 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.272058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq" event={"ID":"c6f723a8-b5fd-4d98-9e48-998873d1acda","Type":"ContainerDied","Data":"b873f39cf2834407eb29116f7ae172ce50943412ab945c8e8f006b5844775b02"}
Jul 29 19:36:53.273353 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.273332    2579 generic.go:356] "Generic (PLEG): container finished" podID="42650d1f-4de2-4db8-810d-f9b9296e1962" containerID="94d7f47a18916b3b124e8e9de47dcf09959226126120258114596ae8401e0e2c" exitCode=0
Jul 29 19:36:53.273449 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.273365    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"
Jul 29 19:36:53.273449 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.273396    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w" event={"ID":"42650d1f-4de2-4db8-810d-f9b9296e1962","Type":"ContainerDied","Data":"94d7f47a18916b3b124e8e9de47dcf09959226126120258114596ae8401e0e2c"}
Jul 29 19:36:53.273449 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.273429    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w" event={"ID":"42650d1f-4de2-4db8-810d-f9b9296e1962","Type":"ContainerDied","Data":"afc9512408b1c53b5a87882bf8522368986e69e86ebec418ed9c2f90096fc9c3"}
Jul 29 19:36:53.281558 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.281542    2579 scope.go:117] "RemoveContainer" containerID="b2c1a947ecfb31f3bb97664629731defc79f29636b4d1cf53640b1a0a6ef58ef"
Jul 29 19:36:53.281820 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:36:53.281802    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c1a947ecfb31f3bb97664629731defc79f29636b4d1cf53640b1a0a6ef58ef\": container with ID starting with b2c1a947ecfb31f3bb97664629731defc79f29636b4d1cf53640b1a0a6ef58ef not found: ID does not exist" containerID="b2c1a947ecfb31f3bb97664629731defc79f29636b4d1cf53640b1a0a6ef58ef"
Jul 29 19:36:53.281871 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.281829    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c1a947ecfb31f3bb97664629731defc79f29636b4d1cf53640b1a0a6ef58ef"} err="failed to get container status \"b2c1a947ecfb31f3bb97664629731defc79f29636b4d1cf53640b1a0a6ef58ef\": rpc error: code = NotFound desc = could not find container \"b2c1a947ecfb31f3bb97664629731defc79f29636b4d1cf53640b1a0a6ef58ef\": container with ID starting with b2c1a947ecfb31f3bb97664629731defc79f29636b4d1cf53640b1a0a6ef58ef not found: ID does not exist"
Jul 29 19:36:53.281871 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.281846    2579 scope.go:117] "RemoveContainer" containerID="f9a8e0b1ce7e7577416cb324412938d144486c2c674674302220fee4ff0d5470"
Jul 29 19:36:53.289414 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.289398    2579 scope.go:117] "RemoveContainer" containerID="f9a8e0b1ce7e7577416cb324412938d144486c2c674674302220fee4ff0d5470"
Jul 29 19:36:53.289649 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:36:53.289633    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a8e0b1ce7e7577416cb324412938d144486c2c674674302220fee4ff0d5470\": container with ID starting with f9a8e0b1ce7e7577416cb324412938d144486c2c674674302220fee4ff0d5470 not found: ID does not exist" containerID="f9a8e0b1ce7e7577416cb324412938d144486c2c674674302220fee4ff0d5470"
Jul 29 19:36:53.289697 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.289654    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a8e0b1ce7e7577416cb324412938d144486c2c674674302220fee4ff0d5470"} err="failed to get container status \"f9a8e0b1ce7e7577416cb324412938d144486c2c674674302220fee4ff0d5470\": rpc error: code = NotFound desc = could not find container \"f9a8e0b1ce7e7577416cb324412938d144486c2c674674302220fee4ff0d5470\": container with ID starting with f9a8e0b1ce7e7577416cb324412938d144486c2c674674302220fee4ff0d5470 not found: ID does not exist"
Jul 29 19:36:53.289697 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.289667    2579 scope.go:117] "RemoveContainer" containerID="94d7f47a18916b3b124e8e9de47dcf09959226126120258114596ae8401e0e2c"
Jul 29 19:36:53.296769 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.296683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-maas-api-tls\") pod \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\" (UID: \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\") "
Jul 29 19:36:53.296769 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.296724    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m978k\" (UniqueName: \"kubernetes.io/projected/c6f723a8-b5fd-4d98-9e48-998873d1acda-kube-api-access-m978k\") pod \"c6f723a8-b5fd-4d98-9e48-998873d1acda\" (UID: \"c6f723a8-b5fd-4d98-9e48-998873d1acda\") "
Jul 29 19:36:53.296769 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.296753    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-openshift-service-ca\") pod \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\" (UID: \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\") "
Jul 29 19:36:53.296957 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.296792    2579 scope.go:117] "RemoveContainer" containerID="94d7f47a18916b3b124e8e9de47dcf09959226126120258114596ae8401e0e2c"
Jul 29 19:36:53.296957 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.296799    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f723a8-b5fd-4d98-9e48-998873d1acda-plugins-config-volume\") pod \"c6f723a8-b5fd-4d98-9e48-998873d1acda\" (UID: \"c6f723a8-b5fd-4d98-9e48-998873d1acda\") "
Jul 29 19:36:53.296957 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.296835    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42650d1f-4de2-4db8-810d-f9b9296e1962-plugins-config-volume\") pod \"42650d1f-4de2-4db8-810d-f9b9296e1962\" (UID: \"42650d1f-4de2-4db8-810d-f9b9296e1962\") "
Jul 29 19:36:53.296957 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.296866    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj5sp\" (UniqueName: \"kubernetes.io/projected/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-kube-api-access-gj5sp\") pod \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\" (UID: \"3ed8ac6d-17da-4948-b587-9a41bda5e0e8\") "
Jul 29 19:36:53.296957 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.296954    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hftqt\" (UniqueName: \"kubernetes.io/projected/42650d1f-4de2-4db8-810d-f9b9296e1962-kube-api-access-hftqt\") pod \"42650d1f-4de2-4db8-810d-f9b9296e1962\" (UID: \"42650d1f-4de2-4db8-810d-f9b9296e1962\") "
Jul 29 19:36:53.297236 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.297117    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ed8ac6d-17da-4948-b587-9a41bda5e0e8" (UID: "3ed8ac6d-17da-4948-b587-9a41bda5e0e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:36:53.297236 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.297180    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42650d1f-4de2-4db8-810d-f9b9296e1962-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42650d1f-4de2-4db8-810d-f9b9296e1962" (UID: "42650d1f-4de2-4db8-810d-f9b9296e1962"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:36:53.297346 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:36:53.297239    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d7f47a18916b3b124e8e9de47dcf09959226126120258114596ae8401e0e2c\": container with ID starting with 94d7f47a18916b3b124e8e9de47dcf09959226126120258114596ae8401e0e2c not found: ID does not exist" containerID="94d7f47a18916b3b124e8e9de47dcf09959226126120258114596ae8401e0e2c"
Jul 29 19:36:53.297346 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.297270    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d7f47a18916b3b124e8e9de47dcf09959226126120258114596ae8401e0e2c"} err="failed to get container status \"94d7f47a18916b3b124e8e9de47dcf09959226126120258114596ae8401e0e2c\": rpc error: code = NotFound desc = could not find container \"94d7f47a18916b3b124e8e9de47dcf09959226126120258114596ae8401e0e2c\": container with ID starting with 94d7f47a18916b3b124e8e9de47dcf09959226126120258114596ae8401e0e2c not found: ID does not exist"
Jul 29 19:36:53.297346 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.297288    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42650d1f-4de2-4db8-810d-f9b9296e1962-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:36:53.297493 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.297308    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:36:53.297642 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.297622    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f723a8-b5fd-4d98-9e48-998873d1acda-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c6f723a8-b5fd-4d98-9e48-998873d1acda" (UID: "c6f723a8-b5fd-4d98-9e48-998873d1acda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:36:53.299041 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.299016    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f723a8-b5fd-4d98-9e48-998873d1acda-kube-api-access-m978k" (OuterVolumeSpecName: "kube-api-access-m978k") pod "c6f723a8-b5fd-4d98-9e48-998873d1acda" (UID: "c6f723a8-b5fd-4d98-9e48-998873d1acda"). InnerVolumeSpecName "kube-api-access-m978k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:36:53.299041 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.299025    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ed8ac6d-17da-4948-b587-9a41bda5e0e8" (UID: "3ed8ac6d-17da-4948-b587-9a41bda5e0e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:36:53.299233 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.299181    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-kube-api-access-gj5sp" (OuterVolumeSpecName: "kube-api-access-gj5sp") pod "3ed8ac6d-17da-4948-b587-9a41bda5e0e8" (UID: "3ed8ac6d-17da-4948-b587-9a41bda5e0e8"). InnerVolumeSpecName "kube-api-access-gj5sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:36:53.299503 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.299485    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42650d1f-4de2-4db8-810d-f9b9296e1962-kube-api-access-hftqt" (OuterVolumeSpecName: "kube-api-access-hftqt") pod "42650d1f-4de2-4db8-810d-f9b9296e1962" (UID: "42650d1f-4de2-4db8-810d-f9b9296e1962"). InnerVolumeSpecName "kube-api-access-hftqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:36:53.398283 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.398265    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m978k\" (UniqueName: \"kubernetes.io/projected/c6f723a8-b5fd-4d98-9e48-998873d1acda-kube-api-access-m978k\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:36:53.398283 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.398284    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f723a8-b5fd-4d98-9e48-998873d1acda-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:36:53.398415 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.398294    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj5sp\" (UniqueName: \"kubernetes.io/projected/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-kube-api-access-gj5sp\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:36:53.398415 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.398302    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hftqt\" (UniqueName: \"kubernetes.io/projected/42650d1f-4de2-4db8-810d-f9b9296e1962-kube-api-access-hftqt\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:36:53.398415 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.398311    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed8ac6d-17da-4948-b587-9a41bda5e0e8-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:36:53.602245 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.602224    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"]
Jul 29 19:36:53.609077 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.609058    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7beba8e5-f6b5f5ff6-rsj7w"]
Jul 29 19:36:53.630486 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.630467    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"]
Jul 29 19:36:53.634590 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.634569    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7beba8e5-c8f79cd79-84z52"]
Jul 29 19:36:53.665600 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.665549    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"]
Jul 29 19:36:53.671473 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:53.671456    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq"]
Jul 29 19:36:54.073225 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:54.073128    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-7beba8e5-669d6bc5cd-ln5cq" podUID="c6f723a8-b5fd-4d98-9e48-998873d1acda" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.57:9004: i/o timeout"
Jul 29 19:36:54.195237 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:54.195208    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ed8ac6d-17da-4948-b587-9a41bda5e0e8" path="/var/lib/kubelet/pods/3ed8ac6d-17da-4948-b587-9a41bda5e0e8/volumes"
Jul 29 19:36:54.195533 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:54.195521    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42650d1f-4de2-4db8-810d-f9b9296e1962" path="/var/lib/kubelet/pods/42650d1f-4de2-4db8-810d-f9b9296e1962/volumes"
Jul 29 19:36:54.195831 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:36:54.195819    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6f723a8-b5fd-4d98-9e48-998873d1acda" path="/var/lib/kubelet/pods/c6f723a8-b5fd-4d98-9e48-998873d1acda/volumes"
Jul 29 19:37:22.350671 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.350630    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"]
Jul 29 19:37:22.351060 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.351036    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed8ac6d-17da-4948-b587-9a41bda5e0e8" containerName="maas-api"
Jul 29 19:37:22.351060 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.351048    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed8ac6d-17da-4948-b587-9a41bda5e0e8" containerName="maas-api"
Jul 29 19:37:22.351163 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.351076    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42650d1f-4de2-4db8-810d-f9b9296e1962" containerName="payload-pre-processing"
Jul 29 19:37:22.351163 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.351082    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="42650d1f-4de2-4db8-810d-f9b9296e1962" containerName="payload-pre-processing"
Jul 29 19:37:22.351163 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.351090    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f723a8-b5fd-4d98-9e48-998873d1acda" containerName="payload-processing"
Jul 29 19:37:22.351163 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.351095    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f723a8-b5fd-4d98-9e48-998873d1acda" containerName="payload-processing"
Jul 29 19:37:22.351337 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.351164    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed8ac6d-17da-4948-b587-9a41bda5e0e8" containerName="maas-api"
Jul 29 19:37:22.351337 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.351176    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6f723a8-b5fd-4d98-9e48-998873d1acda" containerName="payload-processing"
Jul 29 19:37:22.351337 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.351204    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="42650d1f-4de2-4db8-810d-f9b9296e1962" containerName="payload-pre-processing"
Jul 29 19:37:22.354517 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.354485    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:37:22.357586 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.357560    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:37:22.358830 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.358810    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:37:22.358830 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.358824    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-77b3a01f-serving-cert\""
Jul 29 19:37:22.359071 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.358826    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5bzmf\""
Jul 29 19:37:22.366843 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.366812    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"]
Jul 29 19:37:22.447314 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.447273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16590ed6-647d-42ea-add8-d8fbf93e77df-maas-api-tls\") pod \"maas-api-e2e-mt-77b3a01f-78d54f55c7-82872\" (UID: \"16590ed6-647d-42ea-add8-d8fbf93e77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:37:22.447519 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.447401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtpx\" (UniqueName: \"kubernetes.io/projected/16590ed6-647d-42ea-add8-d8fbf93e77df-kube-api-access-lbtpx\") pod \"maas-api-e2e-mt-77b3a01f-78d54f55c7-82872\" (UID: \"16590ed6-647d-42ea-add8-d8fbf93e77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:37:22.447519 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.447463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16590ed6-647d-42ea-add8-d8fbf93e77df-openshift-service-ca\") pod \"maas-api-e2e-mt-77b3a01f-78d54f55c7-82872\" (UID: \"16590ed6-647d-42ea-add8-d8fbf93e77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:37:22.548751 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.548717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16590ed6-647d-42ea-add8-d8fbf93e77df-maas-api-tls\") pod \"maas-api-e2e-mt-77b3a01f-78d54f55c7-82872\" (UID: \"16590ed6-647d-42ea-add8-d8fbf93e77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:37:22.548946 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.548794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtpx\" (UniqueName: \"kubernetes.io/projected/16590ed6-647d-42ea-add8-d8fbf93e77df-kube-api-access-lbtpx\") pod \"maas-api-e2e-mt-77b3a01f-78d54f55c7-82872\" (UID: \"16590ed6-647d-42ea-add8-d8fbf93e77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:37:22.548946 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.548850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16590ed6-647d-42ea-add8-d8fbf93e77df-openshift-service-ca\") pod \"maas-api-e2e-mt-77b3a01f-78d54f55c7-82872\" (UID: \"16590ed6-647d-42ea-add8-d8fbf93e77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:37:22.548946 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:37:22.548882    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-serving-cert: secret "maas-api-e2e-mt-77b3a01f-serving-cert" not found
Jul 29 19:37:22.549113 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:37:22.548968    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16590ed6-647d-42ea-add8-d8fbf93e77df-maas-api-tls podName:16590ed6-647d-42ea-add8-d8fbf93e77df nodeName:}" failed. No retries permitted until 2026-07-29 19:37:23.048943785 +0000 UTC m=+2209.348616268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16590ed6-647d-42ea-add8-d8fbf93e77df-maas-api-tls") pod "maas-api-e2e-mt-77b3a01f-78d54f55c7-82872" (UID: "16590ed6-647d-42ea-add8-d8fbf93e77df") : secret "maas-api-e2e-mt-77b3a01f-serving-cert" not found
Jul 29 19:37:22.549518 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.549499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16590ed6-647d-42ea-add8-d8fbf93e77df-openshift-service-ca\") pod \"maas-api-e2e-mt-77b3a01f-78d54f55c7-82872\" (UID: \"16590ed6-647d-42ea-add8-d8fbf93e77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:37:22.558619 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:22.558587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtpx\" (UniqueName: \"kubernetes.io/projected/16590ed6-647d-42ea-add8-d8fbf93e77df-kube-api-access-lbtpx\") pod \"maas-api-e2e-mt-77b3a01f-78d54f55c7-82872\" (UID: \"16590ed6-647d-42ea-add8-d8fbf93e77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:37:23.052890 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.052854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16590ed6-647d-42ea-add8-d8fbf93e77df-maas-api-tls\") pod \"maas-api-e2e-mt-77b3a01f-78d54f55c7-82872\" (UID: \"16590ed6-647d-42ea-add8-d8fbf93e77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:37:23.055548 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.055521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16590ed6-647d-42ea-add8-d8fbf93e77df-maas-api-tls\") pod \"maas-api-e2e-mt-77b3a01f-78d54f55c7-82872\" (UID: \"16590ed6-647d-42ea-add8-d8fbf93e77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:37:23.074911 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.074874    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"]
Jul 29 19:37:23.078649 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.078622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"
Jul 29 19:37:23.081994 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.081972    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-77b3a01f\""
Jul 29 19:37:23.082134 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.082076    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-77b3a01f-dockercfg-q7mc4\""
Jul 29 19:37:23.089255 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.089226    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"]
Jul 29 19:37:23.123504 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.123466    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"]
Jul 29 19:37:23.127113 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.127095    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"
Jul 29 19:37:23.139119 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.139088    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"]
Jul 29 19:37:23.154077 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.154046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvnf\" (UniqueName: \"kubernetes.io/projected/907111d3-9ea6-49d9-8495-ddb33b44f158-kube-api-access-mxvnf\") pod \"payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc\" (UID: \"907111d3-9ea6-49d9-8495-ddb33b44f158\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"
Jul 29 19:37:23.154265 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.154096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt4gb\" (UniqueName: \"kubernetes.io/projected/a388b4b3-fc61-4501-b68a-9ae6790072e9-kube-api-access-kt4gb\") pod \"payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq\" (UID: \"a388b4b3-fc61-4501-b68a-9ae6790072e9\") " pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"
Jul 29 19:37:23.154265 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.154158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a388b4b3-fc61-4501-b68a-9ae6790072e9-plugins-config-volume\") pod \"payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq\" (UID: \"a388b4b3-fc61-4501-b68a-9ae6790072e9\") " pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"
Jul 29 19:37:23.154353 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.154292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907111d3-9ea6-49d9-8495-ddb33b44f158-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc\" (UID: \"907111d3-9ea6-49d9-8495-ddb33b44f158\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"
Jul 29 19:37:23.255547 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.255509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907111d3-9ea6-49d9-8495-ddb33b44f158-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc\" (UID: \"907111d3-9ea6-49d9-8495-ddb33b44f158\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"
Jul 29 19:37:23.255547 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.255555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvnf\" (UniqueName: \"kubernetes.io/projected/907111d3-9ea6-49d9-8495-ddb33b44f158-kube-api-access-mxvnf\") pod \"payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc\" (UID: \"907111d3-9ea6-49d9-8495-ddb33b44f158\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"
Jul 29 19:37:23.255823 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.255600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt4gb\" (UniqueName: \"kubernetes.io/projected/a388b4b3-fc61-4501-b68a-9ae6790072e9-kube-api-access-kt4gb\") pod \"payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq\" (UID: \"a388b4b3-fc61-4501-b68a-9ae6790072e9\") " pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"
Jul 29 19:37:23.255823 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.255714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a388b4b3-fc61-4501-b68a-9ae6790072e9-plugins-config-volume\") pod \"payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq\" (UID: \"a388b4b3-fc61-4501-b68a-9ae6790072e9\") " pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"
Jul 29 19:37:23.256298 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.256278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907111d3-9ea6-49d9-8495-ddb33b44f158-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc\" (UID: \"907111d3-9ea6-49d9-8495-ddb33b44f158\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"
Jul 29 19:37:23.256453 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.256437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a388b4b3-fc61-4501-b68a-9ae6790072e9-plugins-config-volume\") pod \"payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq\" (UID: \"a388b4b3-fc61-4501-b68a-9ae6790072e9\") " pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"
Jul 29 19:37:23.265702 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.265678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt4gb\" (UniqueName: \"kubernetes.io/projected/a388b4b3-fc61-4501-b68a-9ae6790072e9-kube-api-access-kt4gb\") pod \"payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq\" (UID: \"a388b4b3-fc61-4501-b68a-9ae6790072e9\") " pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"
Jul 29 19:37:23.265827 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.265772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvnf\" (UniqueName: \"kubernetes.io/projected/907111d3-9ea6-49d9-8495-ddb33b44f158-kube-api-access-mxvnf\") pod \"payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc\" (UID: \"907111d3-9ea6-49d9-8495-ddb33b44f158\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"
Jul 29 19:37:23.267577 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.267557    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:37:23.391316 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.391272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"
Jul 29 19:37:23.410886 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.410858    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"]
Jul 29 19:37:23.412687 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:37:23.412658    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16590ed6_647d_42ea_add8_d8fbf93e77df.slice/crio-6bcf9603aec430f5dac7b421f2c3166a42d0f508bd4f5ac026149d3d2c42df7e WatchSource:0}: Error finding container 6bcf9603aec430f5dac7b421f2c3166a42d0f508bd4f5ac026149d3d2c42df7e: Status 404 returned error can't find the container with id 6bcf9603aec430f5dac7b421f2c3166a42d0f508bd4f5ac026149d3d2c42df7e
Jul 29 19:37:23.414253 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.414224    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:37:23.422205 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.422160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872" event={"ID":"16590ed6-647d-42ea-add8-d8fbf93e77df","Type":"ContainerStarted","Data":"6bcf9603aec430f5dac7b421f2c3166a42d0f508bd4f5ac026149d3d2c42df7e"}
Jul 29 19:37:23.438481 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.438451    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"
Jul 29 19:37:23.546255 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.546222    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"]
Jul 29 19:37:23.548351 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:37:23.548318    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda388b4b3_fc61_4501_b68a_9ae6790072e9.slice/crio-924afee978bb79f5aa16e9457e81870b04bb1d3b43992ee14b18462ffb44a110 WatchSource:0}: Error finding container 924afee978bb79f5aa16e9457e81870b04bb1d3b43992ee14b18462ffb44a110: Status 404 returned error can't find the container with id 924afee978bb79f5aa16e9457e81870b04bb1d3b43992ee14b18462ffb44a110
Jul 29 19:37:23.601956 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:23.601929    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"]
Jul 29 19:37:23.605515 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:37:23.605491    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907111d3_9ea6_49d9_8495_ddb33b44f158.slice/crio-c9c13d0720735347a7a8807f5d6045043f8bb3a65b38d3cf5d4e9e5ba95907a4 WatchSource:0}: Error finding container c9c13d0720735347a7a8807f5d6045043f8bb3a65b38d3cf5d4e9e5ba95907a4: Status 404 returned error can't find the container with id c9c13d0720735347a7a8807f5d6045043f8bb3a65b38d3cf5d4e9e5ba95907a4
Jul 29 19:37:24.428852 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:24.428818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc" event={"ID":"907111d3-9ea6-49d9-8495-ddb33b44f158","Type":"ContainerStarted","Data":"94ad11209d4e9c64ce04df2e77c99720dbf8ae6ff7bafe2d1e31d516582cc841"}
Jul 29 19:37:24.429379 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:24.428883    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc" event={"ID":"907111d3-9ea6-49d9-8495-ddb33b44f158","Type":"ContainerStarted","Data":"c9c13d0720735347a7a8807f5d6045043f8bb3a65b38d3cf5d4e9e5ba95907a4"}
Jul 29 19:37:24.429379 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:24.428913    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"
Jul 29 19:37:24.430376 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:24.430353    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872" event={"ID":"16590ed6-647d-42ea-add8-d8fbf93e77df","Type":"ContainerStarted","Data":"767e1f75eb10222e01051ef6f7689e0da3e21c45615a59db1b3c966f7d360223"}
Jul 29 19:37:24.430502 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:24.430408    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:37:24.431868 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:24.431840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq" event={"ID":"a388b4b3-fc61-4501-b68a-9ae6790072e9","Type":"ContainerStarted","Data":"8bebc375b91687ea37ace8d672bd9b572bece0035685c11154ca09bc424408ad"}
Jul 29 19:37:24.431947 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:24.431876    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq" event={"ID":"a388b4b3-fc61-4501-b68a-9ae6790072e9","Type":"ContainerStarted","Data":"924afee978bb79f5aa16e9457e81870b04bb1d3b43992ee14b18462ffb44a110"}
Jul 29 19:37:24.431947 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:24.431897    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"
Jul 29 19:37:24.447335 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:24.447276    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc" podStartSLOduration=1.447260164 podStartE2EDuration="1.447260164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:37:24.445384593 +0000 UTC m=+2210.745057093" watchObservedRunningTime="2026-07-29 19:37:24.447260164 +0000 UTC m=+2210.746932663"
Jul 29 19:37:24.464261 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:24.464184    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872" podStartSLOduration=2.005029701 podStartE2EDuration="2.464167864s" totalImagesPullingTime="459.138163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:37:24.462537633 +0000 UTC m=+2210.762210117" watchObservedRunningTime="2026-07-29 19:37:24.464167864 +0000 UTC m=+2210.763840363"
Jul 29 19:37:24.480055 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:24.479999    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq" podStartSLOduration=1.47998108 podStartE2EDuration="1.47998108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:37:24.478827687 +0000 UTC m=+2210.778500200" watchObservedRunningTime="2026-07-29 19:37:24.47998108 +0000 UTC m=+2210.779653579"
Jul 29 19:37:30.442591 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:30.442562    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:37:35.440045 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:35.440013    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"
Jul 29 19:37:35.440543 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:37:35.440094    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"
Jul 29 19:38:05.104232 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.104133    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"]
Jul 29 19:38:05.108118 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.108102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.111139 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.111104    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-77b3a01f\"/\"kube-root-ca.crt\""
Jul 29 19:38:05.111445 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.111424    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-77b3a01f\"/\"openshift-service-ca.crt\""
Jul 29 19:38:05.112557 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.112540    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-77b3a01f\"/\"default-dockercfg-gjbg5\""
Jul 29 19:38:05.112781 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.112766    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-77b3a01f\"/\"e2e-lifecycle-model-77b3a01f-kserve-self-signed-certs\""
Jul 29 19:38:05.117614 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.117592    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"]
Jul 29 19:38:05.160686 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.160645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-model-cache\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.160686 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.160682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd01518-7e09-4899-be9b-56466517f7f5-tls-certs\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.160892 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.160699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-dshm\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.160892 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.160714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-tmp-dir\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.160892 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.160834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-home\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.160991 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.160905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kghqd\" (UniqueName: \"kubernetes.io/projected/3bd01518-7e09-4899-be9b-56466517f7f5-kube-api-access-kghqd\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.262370 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.262331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-home\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.262557 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.262401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kghqd\" (UniqueName: \"kubernetes.io/projected/3bd01518-7e09-4899-be9b-56466517f7f5-kube-api-access-kghqd\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.262557 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.262427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-model-cache\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.262557 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.262442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd01518-7e09-4899-be9b-56466517f7f5-tls-certs\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.262557 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.262461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-dshm\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.262557 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.262485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-tmp-dir\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.262822 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.262798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-home\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.262866 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.262820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-model-cache\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.262920 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.262904    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-tmp-dir\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.264825 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.264805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-dshm\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.265006 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.264990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd01518-7e09-4899-be9b-56466517f7f5-tls-certs\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.272138 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.272107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kghqd\" (UniqueName: \"kubernetes.io/projected/3bd01518-7e09-4899-be9b-56466517f7f5-kube-api-access-kghqd\") pod \"e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") " pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.417269 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.417234    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:05.559389 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.559366    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"]
Jul 29 19:38:05.560952 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:38:05.560919    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bd01518_7e09_4899_be9b_56466517f7f5.slice/crio-b5a9cedf4dbedc981c97af51ba89deb025ac6caa68d176a17e51c648f10e43e9 WatchSource:0}: Error finding container b5a9cedf4dbedc981c97af51ba89deb025ac6caa68d176a17e51c648f10e43e9: Status 404 returned error can't find the container with id b5a9cedf4dbedc981c97af51ba89deb025ac6caa68d176a17e51c648f10e43e9
Jul 29 19:38:05.639614 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.639575    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln" event={"ID":"3bd01518-7e09-4899-be9b-56466517f7f5","Type":"ContainerStarted","Data":"1e8d8869e9f37001f60ef5fc51a30bc5efc5ca372dd1b9b4d4c8fbc7fc18eb0f"}
Jul 29 19:38:05.639614 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.639619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln" event={"ID":"3bd01518-7e09-4899-be9b-56466517f7f5","Type":"ContainerStarted","Data":"b5a9cedf4dbedc981c97af51ba89deb025ac6caa68d176a17e51c648f10e43e9"}
Jul 29 19:38:05.660934 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:05.660884    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln" podStartSLOduration=0.660869274 podStartE2EDuration="660.869274ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:38:05.659400084 +0000 UTC m=+2251.959072585" watchObservedRunningTime="2026-07-29 19:38:05.660869274 +0000 UTC m=+2251.960541774"
Jul 29 19:38:15.417932 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:15.417890    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:15.418433 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:15.417952    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:15.430283 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:15.430258    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:15.702037 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:15.701951    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:38:34.358591 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:34.358559    2579 scope.go:117] "RemoveContainer" containerID="8fb34a912a7ef3edd268301e29a0ce539e96d70189c0ad02ebd0ac540e10a065"
Jul 29 19:38:42.107478 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.107440    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"]
Jul 29 19:38:42.107890 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.107653    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872" podUID="16590ed6-647d-42ea-add8-d8fbf93e77df" containerName="maas-api" containerID="cri-o://767e1f75eb10222e01051ef6f7689e0da3e21c45615a59db1b3c966f7d360223" gracePeriod=30
Jul 29 19:38:42.153760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.153727    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"]
Jul 29 19:38:42.153957 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.153932    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq" podUID="a388b4b3-fc61-4501-b68a-9ae6790072e9" containerName="payload-processing" containerID="cri-o://8bebc375b91687ea37ace8d672bd9b572bece0035685c11154ca09bc424408ad" gracePeriod=30
Jul 29 19:38:42.163216 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.163166    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"]
Jul 29 19:38:42.163691 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.163655    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc" podUID="907111d3-9ea6-49d9-8495-ddb33b44f158" containerName="payload-pre-processing" containerID="cri-o://94ad11209d4e9c64ce04df2e77c99720dbf8ae6ff7bafe2d1e31d516582cc841" gracePeriod=30
Jul 29 19:38:42.435080 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.435053    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:38:42.437829 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.437809    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"
Jul 29 19:38:42.453070 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.453044    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"
Jul 29 19:38:42.510396 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.510364    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907111d3-9ea6-49d9-8495-ddb33b44f158-plugins-config-volume\") pod \"907111d3-9ea6-49d9-8495-ddb33b44f158\" (UID: \"907111d3-9ea6-49d9-8495-ddb33b44f158\") "
Jul 29 19:38:42.510550 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.510422    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16590ed6-647d-42ea-add8-d8fbf93e77df-maas-api-tls\") pod \"16590ed6-647d-42ea-add8-d8fbf93e77df\" (UID: \"16590ed6-647d-42ea-add8-d8fbf93e77df\") "
Jul 29 19:38:42.510550 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.510447    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxvnf\" (UniqueName: \"kubernetes.io/projected/907111d3-9ea6-49d9-8495-ddb33b44f158-kube-api-access-mxvnf\") pod \"907111d3-9ea6-49d9-8495-ddb33b44f158\" (UID: \"907111d3-9ea6-49d9-8495-ddb33b44f158\") "
Jul 29 19:38:42.510550 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.510490    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbtpx\" (UniqueName: \"kubernetes.io/projected/16590ed6-647d-42ea-add8-d8fbf93e77df-kube-api-access-lbtpx\") pod \"16590ed6-647d-42ea-add8-d8fbf93e77df\" (UID: \"16590ed6-647d-42ea-add8-d8fbf93e77df\") "
Jul 29 19:38:42.510550 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.510530    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16590ed6-647d-42ea-add8-d8fbf93e77df-openshift-service-ca\") pod \"16590ed6-647d-42ea-add8-d8fbf93e77df\" (UID: \"16590ed6-647d-42ea-add8-d8fbf93e77df\") "
Jul 29 19:38:42.510777 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.510555    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a388b4b3-fc61-4501-b68a-9ae6790072e9-plugins-config-volume\") pod \"a388b4b3-fc61-4501-b68a-9ae6790072e9\" (UID: \"a388b4b3-fc61-4501-b68a-9ae6790072e9\") "
Jul 29 19:38:42.510777 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.510580    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt4gb\" (UniqueName: \"kubernetes.io/projected/a388b4b3-fc61-4501-b68a-9ae6790072e9-kube-api-access-kt4gb\") pod \"a388b4b3-fc61-4501-b68a-9ae6790072e9\" (UID: \"a388b4b3-fc61-4501-b68a-9ae6790072e9\") "
Jul 29 19:38:42.510906 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.510878    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907111d3-9ea6-49d9-8495-ddb33b44f158-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "907111d3-9ea6-49d9-8495-ddb33b44f158" (UID: "907111d3-9ea6-49d9-8495-ddb33b44f158"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:38:42.510965 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.510922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16590ed6-647d-42ea-add8-d8fbf93e77df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16590ed6-647d-42ea-add8-d8fbf93e77df" (UID: "16590ed6-647d-42ea-add8-d8fbf93e77df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:38:42.511375 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.511344    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a388b4b3-fc61-4501-b68a-9ae6790072e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a388b4b3-fc61-4501-b68a-9ae6790072e9" (UID: "a388b4b3-fc61-4501-b68a-9ae6790072e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:38:42.512823 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.512797    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16590ed6-647d-42ea-add8-d8fbf93e77df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16590ed6-647d-42ea-add8-d8fbf93e77df" (UID: "16590ed6-647d-42ea-add8-d8fbf93e77df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:38:42.512906 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.512847    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907111d3-9ea6-49d9-8495-ddb33b44f158-kube-api-access-mxvnf" (OuterVolumeSpecName: "kube-api-access-mxvnf") pod "907111d3-9ea6-49d9-8495-ddb33b44f158" (UID: "907111d3-9ea6-49d9-8495-ddb33b44f158"). InnerVolumeSpecName "kube-api-access-mxvnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:38:42.512906 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.512868    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16590ed6-647d-42ea-add8-d8fbf93e77df-kube-api-access-lbtpx" (OuterVolumeSpecName: "kube-api-access-lbtpx") pod "16590ed6-647d-42ea-add8-d8fbf93e77df" (UID: "16590ed6-647d-42ea-add8-d8fbf93e77df"). InnerVolumeSpecName "kube-api-access-lbtpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:38:42.513158 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.513137    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a388b4b3-fc61-4501-b68a-9ae6790072e9-kube-api-access-kt4gb" (OuterVolumeSpecName: "kube-api-access-kt4gb") pod "a388b4b3-fc61-4501-b68a-9ae6790072e9" (UID: "a388b4b3-fc61-4501-b68a-9ae6790072e9"). InnerVolumeSpecName "kube-api-access-kt4gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:38:42.611964 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.611923    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907111d3-9ea6-49d9-8495-ddb33b44f158-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:38:42.611964 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.611956    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16590ed6-647d-42ea-add8-d8fbf93e77df-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:38:42.611964 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.611966    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxvnf\" (UniqueName: \"kubernetes.io/projected/907111d3-9ea6-49d9-8495-ddb33b44f158-kube-api-access-mxvnf\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:38:42.611964 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.611975    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbtpx\" (UniqueName: \"kubernetes.io/projected/16590ed6-647d-42ea-add8-d8fbf93e77df-kube-api-access-lbtpx\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:38:42.612267 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.611985    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16590ed6-647d-42ea-add8-d8fbf93e77df-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:38:42.612267 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.611994    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a388b4b3-fc61-4501-b68a-9ae6790072e9-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:38:42.612267 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.612002    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt4gb\" (UniqueName: \"kubernetes.io/projected/a388b4b3-fc61-4501-b68a-9ae6790072e9-kube-api-access-kt4gb\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:38:42.834960 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.834860    2579 generic.go:356] "Generic (PLEG): container finished" podID="907111d3-9ea6-49d9-8495-ddb33b44f158" containerID="94ad11209d4e9c64ce04df2e77c99720dbf8ae6ff7bafe2d1e31d516582cc841" exitCode=0
Jul 29 19:38:42.834960 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.834905    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"
Jul 29 19:38:42.834960 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.834930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc" event={"ID":"907111d3-9ea6-49d9-8495-ddb33b44f158","Type":"ContainerDied","Data":"94ad11209d4e9c64ce04df2e77c99720dbf8ae6ff7bafe2d1e31d516582cc841"}
Jul 29 19:38:42.835284 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.834989    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc" event={"ID":"907111d3-9ea6-49d9-8495-ddb33b44f158","Type":"ContainerDied","Data":"c9c13d0720735347a7a8807f5d6045043f8bb3a65b38d3cf5d4e9e5ba95907a4"}
Jul 29 19:38:42.835284 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.835019    2579 scope.go:117] "RemoveContainer" containerID="94ad11209d4e9c64ce04df2e77c99720dbf8ae6ff7bafe2d1e31d516582cc841"
Jul 29 19:38:42.836434 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.836405    2579 generic.go:356] "Generic (PLEG): container finished" podID="16590ed6-647d-42ea-add8-d8fbf93e77df" containerID="767e1f75eb10222e01051ef6f7689e0da3e21c45615a59db1b3c966f7d360223" exitCode=0
Jul 29 19:38:42.836547 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.836470    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"
Jul 29 19:38:42.836547 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.836447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872" event={"ID":"16590ed6-647d-42ea-add8-d8fbf93e77df","Type":"ContainerDied","Data":"767e1f75eb10222e01051ef6f7689e0da3e21c45615a59db1b3c966f7d360223"}
Jul 29 19:38:42.840853 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.837053    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872" event={"ID":"16590ed6-647d-42ea-add8-d8fbf93e77df","Type":"ContainerDied","Data":"6bcf9603aec430f5dac7b421f2c3166a42d0f508bd4f5ac026149d3d2c42df7e"}
Jul 29 19:38:42.846736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.846699    2579 generic.go:356] "Generic (PLEG): container finished" podID="a388b4b3-fc61-4501-b68a-9ae6790072e9" containerID="8bebc375b91687ea37ace8d672bd9b572bece0035685c11154ca09bc424408ad" exitCode=0
Jul 29 19:38:42.846899 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.846767    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"
Jul 29 19:38:42.847009 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.846987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq" event={"ID":"a388b4b3-fc61-4501-b68a-9ae6790072e9","Type":"ContainerDied","Data":"8bebc375b91687ea37ace8d672bd9b572bece0035685c11154ca09bc424408ad"}
Jul 29 19:38:42.847182 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.847159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq" event={"ID":"a388b4b3-fc61-4501-b68a-9ae6790072e9","Type":"ContainerDied","Data":"924afee978bb79f5aa16e9457e81870b04bb1d3b43992ee14b18462ffb44a110"}
Jul 29 19:38:42.852322 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.852303    2579 scope.go:117] "RemoveContainer" containerID="94ad11209d4e9c64ce04df2e77c99720dbf8ae6ff7bafe2d1e31d516582cc841"
Jul 29 19:38:42.852639 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:38:42.852620    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ad11209d4e9c64ce04df2e77c99720dbf8ae6ff7bafe2d1e31d516582cc841\": container with ID starting with 94ad11209d4e9c64ce04df2e77c99720dbf8ae6ff7bafe2d1e31d516582cc841 not found: ID does not exist" containerID="94ad11209d4e9c64ce04df2e77c99720dbf8ae6ff7bafe2d1e31d516582cc841"
Jul 29 19:38:42.852722 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.852647    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ad11209d4e9c64ce04df2e77c99720dbf8ae6ff7bafe2d1e31d516582cc841"} err="failed to get container status \"94ad11209d4e9c64ce04df2e77c99720dbf8ae6ff7bafe2d1e31d516582cc841\": rpc error: code = NotFound desc = could not find container \"94ad11209d4e9c64ce04df2e77c99720dbf8ae6ff7bafe2d1e31d516582cc841\": container with ID starting with 94ad11209d4e9c64ce04df2e77c99720dbf8ae6ff7bafe2d1e31d516582cc841 not found: ID does not exist"
Jul 29 19:38:42.852722 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.852665    2579 scope.go:117] "RemoveContainer" containerID="767e1f75eb10222e01051ef6f7689e0da3e21c45615a59db1b3c966f7d360223"
Jul 29 19:38:42.862710 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.862688    2579 scope.go:117] "RemoveContainer" containerID="767e1f75eb10222e01051ef6f7689e0da3e21c45615a59db1b3c966f7d360223"
Jul 29 19:38:42.863069 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:38:42.863038    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767e1f75eb10222e01051ef6f7689e0da3e21c45615a59db1b3c966f7d360223\": container with ID starting with 767e1f75eb10222e01051ef6f7689e0da3e21c45615a59db1b3c966f7d360223 not found: ID does not exist" containerID="767e1f75eb10222e01051ef6f7689e0da3e21c45615a59db1b3c966f7d360223"
Jul 29 19:38:42.863161 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.863071    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767e1f75eb10222e01051ef6f7689e0da3e21c45615a59db1b3c966f7d360223"} err="failed to get container status \"767e1f75eb10222e01051ef6f7689e0da3e21c45615a59db1b3c966f7d360223\": rpc error: code = NotFound desc = could not find container \"767e1f75eb10222e01051ef6f7689e0da3e21c45615a59db1b3c966f7d360223\": container with ID starting with 767e1f75eb10222e01051ef6f7689e0da3e21c45615a59db1b3c966f7d360223 not found: ID does not exist"
Jul 29 19:38:42.863161 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.863095    2579 scope.go:117] "RemoveContainer" containerID="8bebc375b91687ea37ace8d672bd9b572bece0035685c11154ca09bc424408ad"
Jul 29 19:38:42.865886 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.865861    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"]
Jul 29 19:38:42.872007 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.871972    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77b3a01f-78d54f55c7-82872"]
Jul 29 19:38:42.873917 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.873894    2579 scope.go:117] "RemoveContainer" containerID="8bebc375b91687ea37ace8d672bd9b572bece0035685c11154ca09bc424408ad"
Jul 29 19:38:42.875144 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:38:42.875119    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bebc375b91687ea37ace8d672bd9b572bece0035685c11154ca09bc424408ad\": container with ID starting with 8bebc375b91687ea37ace8d672bd9b572bece0035685c11154ca09bc424408ad not found: ID does not exist" containerID="8bebc375b91687ea37ace8d672bd9b572bece0035685c11154ca09bc424408ad"
Jul 29 19:38:42.875260 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.875153    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bebc375b91687ea37ace8d672bd9b572bece0035685c11154ca09bc424408ad"} err="failed to get container status \"8bebc375b91687ea37ace8d672bd9b572bece0035685c11154ca09bc424408ad\": rpc error: code = NotFound desc = could not find container \"8bebc375b91687ea37ace8d672bd9b572bece0035685c11154ca09bc424408ad\": container with ID starting with 8bebc375b91687ea37ace8d672bd9b572bece0035685c11154ca09bc424408ad not found: ID does not exist"
Jul 29 19:38:42.883598 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.883553    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"]
Jul 29 19:38:42.888071 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.888041    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77b3a01f-7df58699b-f8lqc"]
Jul 29 19:38:42.899701 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.899669    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"]
Jul 29 19:38:42.905737 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:42.905704    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77b3a01f-7f4986b7c8-2qrhq"]
Jul 29 19:38:44.195548 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:44.195514    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16590ed6-647d-42ea-add8-d8fbf93e77df" path="/var/lib/kubelet/pods/16590ed6-647d-42ea-add8-d8fbf93e77df/volumes"
Jul 29 19:38:44.195903 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:44.195862    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="907111d3-9ea6-49d9-8495-ddb33b44f158" path="/var/lib/kubelet/pods/907111d3-9ea6-49d9-8495-ddb33b44f158/volumes"
Jul 29 19:38:44.196155 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:44.196142    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a388b4b3-fc61-4501-b68a-9ae6790072e9" path="/var/lib/kubelet/pods/a388b4b3-fc61-4501-b68a-9ae6790072e9/volumes"
Jul 29 19:38:50.347050 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:38:50.347017    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"]
Jul 29 19:39:05.365830 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.365785    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln" podUID="3bd01518-7e09-4899-be9b-56466517f7f5" containerName="main" containerID="cri-o://1e8d8869e9f37001f60ef5fc51a30bc5efc5ca372dd1b9b4d4c8fbc7fc18eb0f" gracePeriod=45
Jul 29 19:39:05.609657 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.609636    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:39:05.719900 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.719868    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kghqd\" (UniqueName: \"kubernetes.io/projected/3bd01518-7e09-4899-be9b-56466517f7f5-kube-api-access-kghqd\") pod \"3bd01518-7e09-4899-be9b-56466517f7f5\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") "
Jul 29 19:39:05.720062 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.719926    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd01518-7e09-4899-be9b-56466517f7f5-tls-certs\") pod \"3bd01518-7e09-4899-be9b-56466517f7f5\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") "
Jul 29 19:39:05.720062 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.719957    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-tmp-dir\") pod \"3bd01518-7e09-4899-be9b-56466517f7f5\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") "
Jul 29 19:39:05.720175 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.720061    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-home\") pod \"3bd01518-7e09-4899-be9b-56466517f7f5\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") "
Jul 29 19:39:05.720276 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.720185    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3bd01518-7e09-4899-be9b-56466517f7f5" (UID: "3bd01518-7e09-4899-be9b-56466517f7f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:39:05.720276 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.720229    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-model-cache\") pod \"3bd01518-7e09-4899-be9b-56466517f7f5\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") "
Jul 29 19:39:05.720390 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.720278    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-dshm\") pod \"3bd01518-7e09-4899-be9b-56466517f7f5\" (UID: \"3bd01518-7e09-4899-be9b-56466517f7f5\") "
Jul 29 19:39:05.720390 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.720335    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-home" (OuterVolumeSpecName: "home") pod "3bd01518-7e09-4899-be9b-56466517f7f5" (UID: "3bd01518-7e09-4899-be9b-56466517f7f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:39:05.720468 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.720396    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "3bd01518-7e09-4899-be9b-56466517f7f5" (UID: "3bd01518-7e09-4899-be9b-56466517f7f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:39:05.720572 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.720550    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-tmp-dir\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:39:05.720572 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.720566    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-home\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:39:05.720572 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.720574    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-model-cache\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:39:05.722318 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.722300    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd01518-7e09-4899-be9b-56466517f7f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3bd01518-7e09-4899-be9b-56466517f7f5" (UID: "3bd01518-7e09-4899-be9b-56466517f7f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:39:05.722404 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.722344    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-dshm" (OuterVolumeSpecName: "dshm") pod "3bd01518-7e09-4899-be9b-56466517f7f5" (UID: "3bd01518-7e09-4899-be9b-56466517f7f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:39:05.722443 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.722413    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd01518-7e09-4899-be9b-56466517f7f5-kube-api-access-kghqd" (OuterVolumeSpecName: "kube-api-access-kghqd") pod "3bd01518-7e09-4899-be9b-56466517f7f5" (UID: "3bd01518-7e09-4899-be9b-56466517f7f5"). InnerVolumeSpecName "kube-api-access-kghqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:39:05.821649 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.821617    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd01518-7e09-4899-be9b-56466517f7f5-dshm\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:39:05.821649 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.821645    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kghqd\" (UniqueName: \"kubernetes.io/projected/3bd01518-7e09-4899-be9b-56466517f7f5-kube-api-access-kghqd\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:39:05.821649 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.821655    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd01518-7e09-4899-be9b-56466517f7f5-tls-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:39:05.966356 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.966325    2579 generic.go:356] "Generic (PLEG): container finished" podID="3bd01518-7e09-4899-be9b-56466517f7f5" containerID="1e8d8869e9f37001f60ef5fc51a30bc5efc5ca372dd1b9b4d4c8fbc7fc18eb0f" exitCode=0
Jul 29 19:39:05.966538 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.966374    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"
Jul 29 19:39:05.966538 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.966402    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln" event={"ID":"3bd01518-7e09-4899-be9b-56466517f7f5","Type":"ContainerDied","Data":"1e8d8869e9f37001f60ef5fc51a30bc5efc5ca372dd1b9b4d4c8fbc7fc18eb0f"}
Jul 29 19:39:05.966538 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.966455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln" event={"ID":"3bd01518-7e09-4899-be9b-56466517f7f5","Type":"ContainerDied","Data":"b5a9cedf4dbedc981c97af51ba89deb025ac6caa68d176a17e51c648f10e43e9"}
Jul 29 19:39:05.966538 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.966479    2579 scope.go:117] "RemoveContainer" containerID="1e8d8869e9f37001f60ef5fc51a30bc5efc5ca372dd1b9b4d4c8fbc7fc18eb0f"
Jul 29 19:39:05.976273 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.976250    2579 scope.go:117] "RemoveContainer" containerID="1e8d8869e9f37001f60ef5fc51a30bc5efc5ca372dd1b9b4d4c8fbc7fc18eb0f"
Jul 29 19:39:05.976540 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:39:05.976522    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8d8869e9f37001f60ef5fc51a30bc5efc5ca372dd1b9b4d4c8fbc7fc18eb0f\": container with ID starting with 1e8d8869e9f37001f60ef5fc51a30bc5efc5ca372dd1b9b4d4c8fbc7fc18eb0f not found: ID does not exist" containerID="1e8d8869e9f37001f60ef5fc51a30bc5efc5ca372dd1b9b4d4c8fbc7fc18eb0f"
Jul 29 19:39:05.976588 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.976549    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8d8869e9f37001f60ef5fc51a30bc5efc5ca372dd1b9b4d4c8fbc7fc18eb0f"} err="failed to get container status \"1e8d8869e9f37001f60ef5fc51a30bc5efc5ca372dd1b9b4d4c8fbc7fc18eb0f\": rpc error: code = NotFound desc = could not find container \"1e8d8869e9f37001f60ef5fc51a30bc5efc5ca372dd1b9b4d4c8fbc7fc18eb0f\": container with ID starting with 1e8d8869e9f37001f60ef5fc51a30bc5efc5ca372dd1b9b4d4c8fbc7fc18eb0f not found: ID does not exist"
Jul 29 19:39:05.988303 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.988279    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"]
Jul 29 19:39:05.994008 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:05.993989    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-77b3a01f/e2e-lifecycle-model-77b3a01f-kserve-699968476c-2sfln"]
Jul 29 19:39:06.195372 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:06.195333    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bd01518-7e09-4899-be9b-56466517f7f5" path="/var/lib/kubelet/pods/3bd01518-7e09-4899-be9b-56466517f7f5/volumes"
Jul 29 19:39:34.379022 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:39:34.378947    2579 scope.go:117] "RemoveContainer" containerID="dcef783d7a47b546a9f0cc6944e927f9a2403c45e0c1d26a9b2d056028a8b88e"
Jul 29 19:40:34.300021 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:34.299994    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:40:34.311334 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:34.311315    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:40:47.561426 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.561393    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"]
Jul 29 19:40:47.561949 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.561930    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd01518-7e09-4899-be9b-56466517f7f5" containerName="main"
Jul 29 19:40:47.562027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.561953    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd01518-7e09-4899-be9b-56466517f7f5" containerName="main"
Jul 29 19:40:47.562027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.561966    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16590ed6-647d-42ea-add8-d8fbf93e77df" containerName="maas-api"
Jul 29 19:40:47.562027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.561973    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="16590ed6-647d-42ea-add8-d8fbf93e77df" containerName="maas-api"
Jul 29 19:40:47.562027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.561985    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a388b4b3-fc61-4501-b68a-9ae6790072e9" containerName="payload-processing"
Jul 29 19:40:47.562027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.561993    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a388b4b3-fc61-4501-b68a-9ae6790072e9" containerName="payload-processing"
Jul 29 19:40:47.562027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.562004    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="907111d3-9ea6-49d9-8495-ddb33b44f158" containerName="payload-pre-processing"
Jul 29 19:40:47.562027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.562013    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="907111d3-9ea6-49d9-8495-ddb33b44f158" containerName="payload-pre-processing"
Jul 29 19:40:47.562391 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.562113    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="16590ed6-647d-42ea-add8-d8fbf93e77df" containerName="maas-api"
Jul 29 19:40:47.562391 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.562126    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="907111d3-9ea6-49d9-8495-ddb33b44f158" containerName="payload-pre-processing"
Jul 29 19:40:47.562391 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.562138    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a388b4b3-fc61-4501-b68a-9ae6790072e9" containerName="payload-processing"
Jul 29 19:40:47.562391 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.562150    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd01518-7e09-4899-be9b-56466517f7f5" containerName="main"
Jul 29 19:40:47.565310 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.565288    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:40:47.569130 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.569095    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:40:47.569267 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.569169    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-909676-serving-cert\""
Jul 29 19:40:47.569374 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.569359    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5bzmf\""
Jul 29 19:40:47.570413 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.570395    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:40:47.586012 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.585987    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"]
Jul 29 19:40:47.710545 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.710513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khbjq\" (UniqueName: \"kubernetes.io/projected/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-kube-api-access-khbjq\") pod \"maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg\" (UID: \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:40:47.710740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.710556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-openshift-service-ca\") pod \"maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg\" (UID: \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:40:47.710740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.710604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-maas-api-tls\") pod \"maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg\" (UID: \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:40:47.812075 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.811976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khbjq\" (UniqueName: \"kubernetes.io/projected/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-kube-api-access-khbjq\") pod \"maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg\" (UID: \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:40:47.812075 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.812022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-openshift-service-ca\") pod \"maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg\" (UID: \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:40:47.812075 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.812054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-maas-api-tls\") pod \"maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg\" (UID: \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:40:47.812908 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.812886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-openshift-service-ca\") pod \"maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg\" (UID: \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:40:47.814670 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.814652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-maas-api-tls\") pod \"maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg\" (UID: \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:40:47.821499 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.821477    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khbjq\" (UniqueName: \"kubernetes.io/projected/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-kube-api-access-khbjq\") pod \"maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg\" (UID: \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:40:47.877456 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:47.877421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:40:48.013786 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.013756    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"]
Jul 29 19:40:48.014789 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:40:48.014762    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43b01e8_6d2d_4de2_b7bc_3abbff6e4932.slice/crio-600f69636253d44e1920a243111195dc61032403480910663d3395bcbf43b925 WatchSource:0}: Error finding container 600f69636253d44e1920a243111195dc61032403480910663d3395bcbf43b925: Status 404 returned error can't find the container with id 600f69636253d44e1920a243111195dc61032403480910663d3395bcbf43b925
Jul 29 19:40:48.265423 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.265387    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"]
Jul 29 19:40:48.270076 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.270057    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"
Jul 29 19:40:48.280178 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.280160    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-909676\""
Jul 29 19:40:48.280294 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.280229    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-909676-dockercfg-qxkfm\""
Jul 29 19:40:48.286291 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.286266    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"]
Jul 29 19:40:48.323146 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.323106    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"]
Jul 29 19:40:48.326799 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.326777    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"
Jul 29 19:40:48.335843 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.335820    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"]
Jul 29 19:40:48.417564 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.417527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87xs6\" (UniqueName: \"kubernetes.io/projected/bc836c50-a140-48ec-a9b5-0fc0521798d6-kube-api-access-87xs6\") pod \"payload-processing-e2e-inf-a-909676-565844c848-tnjld\" (UID: \"bc836c50-a140-48ec-a9b5-0fc0521798d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"
Jul 29 19:40:48.417751 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.417570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f153453-f8e9-4455-a6d3-8f936ecbdf79-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff\" (UID: \"8f153453-f8e9-4455-a6d3-8f936ecbdf79\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"
Jul 29 19:40:48.417751 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.417666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnwj6\" (UniqueName: \"kubernetes.io/projected/8f153453-f8e9-4455-a6d3-8f936ecbdf79-kube-api-access-nnwj6\") pod \"payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff\" (UID: \"8f153453-f8e9-4455-a6d3-8f936ecbdf79\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"
Jul 29 19:40:48.417751 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.417715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc836c50-a140-48ec-a9b5-0fc0521798d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-909676-565844c848-tnjld\" (UID: \"bc836c50-a140-48ec-a9b5-0fc0521798d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"
Jul 29 19:40:48.453838 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.453803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg" event={"ID":"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932","Type":"ContainerStarted","Data":"600f69636253d44e1920a243111195dc61032403480910663d3395bcbf43b925"}
Jul 29 19:40:48.519047 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.518942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc836c50-a140-48ec-a9b5-0fc0521798d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-909676-565844c848-tnjld\" (UID: \"bc836c50-a140-48ec-a9b5-0fc0521798d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"
Jul 29 19:40:48.519047 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.519048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87xs6\" (UniqueName: \"kubernetes.io/projected/bc836c50-a140-48ec-a9b5-0fc0521798d6-kube-api-access-87xs6\") pod \"payload-processing-e2e-inf-a-909676-565844c848-tnjld\" (UID: \"bc836c50-a140-48ec-a9b5-0fc0521798d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"
Jul 29 19:40:48.519311 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.519089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f153453-f8e9-4455-a6d3-8f936ecbdf79-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff\" (UID: \"8f153453-f8e9-4455-a6d3-8f936ecbdf79\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"
Jul 29 19:40:48.519311 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.519154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnwj6\" (UniqueName: \"kubernetes.io/projected/8f153453-f8e9-4455-a6d3-8f936ecbdf79-kube-api-access-nnwj6\") pod \"payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff\" (UID: \"8f153453-f8e9-4455-a6d3-8f936ecbdf79\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"
Jul 29 19:40:48.519795 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.519774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f153453-f8e9-4455-a6d3-8f936ecbdf79-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff\" (UID: \"8f153453-f8e9-4455-a6d3-8f936ecbdf79\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"
Jul 29 19:40:48.519851 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.519776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc836c50-a140-48ec-a9b5-0fc0521798d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-909676-565844c848-tnjld\" (UID: \"bc836c50-a140-48ec-a9b5-0fc0521798d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"
Jul 29 19:40:48.528627 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.528602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87xs6\" (UniqueName: \"kubernetes.io/projected/bc836c50-a140-48ec-a9b5-0fc0521798d6-kube-api-access-87xs6\") pod \"payload-processing-e2e-inf-a-909676-565844c848-tnjld\" (UID: \"bc836c50-a140-48ec-a9b5-0fc0521798d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"
Jul 29 19:40:48.528627 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.528622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnwj6\" (UniqueName: \"kubernetes.io/projected/8f153453-f8e9-4455-a6d3-8f936ecbdf79-kube-api-access-nnwj6\") pod \"payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff\" (UID: \"8f153453-f8e9-4455-a6d3-8f936ecbdf79\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"
Jul 29 19:40:48.579853 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.579820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"
Jul 29 19:40:48.639129 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.638881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"
Jul 29 19:40:48.733657 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.733634    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"]
Jul 29 19:40:48.735878 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:40:48.735848    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc836c50_a140_48ec_a9b5_0fc0521798d6.slice/crio-8630db1a45adcb3d348f86f466a2ae372d612e79fb7502a0525cfbfcb8543084 WatchSource:0}: Error finding container 8630db1a45adcb3d348f86f466a2ae372d612e79fb7502a0525cfbfcb8543084: Status 404 returned error can't find the container with id 8630db1a45adcb3d348f86f466a2ae372d612e79fb7502a0525cfbfcb8543084
Jul 29 19:40:48.798775 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:48.798749    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"]
Jul 29 19:40:48.800249 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:40:48.800212    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f153453_f8e9_4455_a6d3_8f936ecbdf79.slice/crio-1f0484db49511ab083ae117d13d27bd9c56080b940172d743988a1f41670eef0 WatchSource:0}: Error finding container 1f0484db49511ab083ae117d13d27bd9c56080b940172d743988a1f41670eef0: Status 404 returned error can't find the container with id 1f0484db49511ab083ae117d13d27bd9c56080b940172d743988a1f41670eef0
Jul 29 19:40:49.460607 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:49.460567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff" event={"ID":"8f153453-f8e9-4455-a6d3-8f936ecbdf79","Type":"ContainerStarted","Data":"15111d7e60a28006bd0ac61d1ea305626e935e5d1f1a5ace6526fe01335746af"}
Jul 29 19:40:49.460818 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:49.460628    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff" event={"ID":"8f153453-f8e9-4455-a6d3-8f936ecbdf79","Type":"ContainerStarted","Data":"1f0484db49511ab083ae117d13d27bd9c56080b940172d743988a1f41670eef0"}
Jul 29 19:40:49.460818 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:49.460772    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"
Jul 29 19:40:49.462037 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:49.462012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg" event={"ID":"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932","Type":"ContainerStarted","Data":"f4ef65f7ad310931eb6ed3412d93f7dd7cf710c83c24faafd32a383ff58b1a92"}
Jul 29 19:40:49.462157 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:49.462076    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:40:49.463497 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:49.463477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld" event={"ID":"bc836c50-a140-48ec-a9b5-0fc0521798d6","Type":"ContainerStarted","Data":"ed7192d6b0b309ab75bac032c703178ba8c257059422ac18458ed46ecdc91d11"}
Jul 29 19:40:49.463581 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:49.463511    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld" event={"ID":"bc836c50-a140-48ec-a9b5-0fc0521798d6","Type":"ContainerStarted","Data":"8630db1a45adcb3d348f86f466a2ae372d612e79fb7502a0525cfbfcb8543084"}
Jul 29 19:40:49.463581 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:49.463529    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"
Jul 29 19:40:49.479750 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:49.479702    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff" podStartSLOduration=1.4796837 podStartE2EDuration="1.4796837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:40:49.47878858 +0000 UTC m=+2415.778461079" watchObservedRunningTime="2026-07-29 19:40:49.4796837 +0000 UTC m=+2415.779356199"
Jul 29 19:40:49.501807 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:49.501756    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg" podStartSLOduration=1.835840229 podStartE2EDuration="2.501740136s" totalImagesPullingTime="665.899907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:40:49.500839467 +0000 UTC m=+2415.800511997" watchObservedRunningTime="2026-07-29 19:40:49.501740136 +0000 UTC m=+2415.801412635"
Jul 29 19:40:49.526391 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:49.526342    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld" podStartSLOduration=1.5263260029999999 podStartE2EDuration="1.526326003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:40:49.52501443 +0000 UTC m=+2415.824686925" watchObservedRunningTime="2026-07-29 19:40:49.526326003 +0000 UTC m=+2415.825998503"
Jul 29 19:40:55.477401 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:40:55.477334    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:41:00.474318 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:00.474279    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"
Jul 29 19:41:00.474722 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:00.474376    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"
Jul 29 19:41:15.746506 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:15.746461    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"]
Jul 29 19:41:15.750584 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:15.750564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:41:15.753363 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:15.753340    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b3bf1c-serving-cert\""
Jul 29 19:41:15.762050 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:15.762024    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"]
Jul 29 19:41:15.873477 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:15.873447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs\" (UID: \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:41:15.873477 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:15.873481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29hsk\" (UniqueName: \"kubernetes.io/projected/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-kube-api-access-29hsk\") pod \"maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs\" (UID: \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:41:15.873698 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:15.873554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-maas-api-tls\") pod \"maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs\" (UID: \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:41:15.974383 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:15.974347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-maas-api-tls\") pod \"maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs\" (UID: \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:41:15.974568 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:15.974463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs\" (UID: \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:41:15.974568 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:15.974497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29hsk\" (UniqueName: \"kubernetes.io/projected/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-kube-api-access-29hsk\") pod \"maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs\" (UID: \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:41:15.975170 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:15.975151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs\" (UID: \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:41:15.976897 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:15.976878    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-maas-api-tls\") pod \"maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs\" (UID: \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:41:15.985287 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:15.985265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29hsk\" (UniqueName: \"kubernetes.io/projected/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-kube-api-access-29hsk\") pod \"maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs\" (UID: \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:41:16.064739 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.064647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:41:16.225868 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.225840    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"]
Jul 29 19:41:16.227954 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:41:16.227927    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b643d1_1d19_4d5d_aff3_6822e9c4b096.slice/crio-7d9b1f258ef9289a7dfe2b5ef2f0dd96c81a9285889354655b522bec65a41611 WatchSource:0}: Error finding container 7d9b1f258ef9289a7dfe2b5ef2f0dd96c81a9285889354655b522bec65a41611: Status 404 returned error can't find the container with id 7d9b1f258ef9289a7dfe2b5ef2f0dd96c81a9285889354655b522bec65a41611
Jul 29 19:41:16.607992 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.607955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs" event={"ID":"b5b643d1-1d19-4d5d-aff3-6822e9c4b096","Type":"ContainerStarted","Data":"7d9b1f258ef9289a7dfe2b5ef2f0dd96c81a9285889354655b522bec65a41611"}
Jul 29 19:41:16.700818 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.700783    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"]
Jul 29 19:41:16.703938 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.703920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"
Jul 29 19:41:16.708546 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.708439    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b3bf1c\""
Jul 29 19:41:16.708546 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.708445    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b3bf1c-dockercfg-6tsvr\""
Jul 29 19:41:16.722157 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.722138    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"]
Jul 29 19:41:16.769730 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.769705    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"]
Jul 29 19:41:16.773059 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.773043    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"
Jul 29 19:41:16.783371 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.783349    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"]
Jul 29 19:41:16.783543 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.783520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d67k5\" (UniqueName: \"kubernetes.io/projected/d1f4bb8a-3175-40f1-aa8a-e6411eb7231b-kube-api-access-d67k5\") pod \"payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r\" (UID: \"d1f4bb8a-3175-40f1-aa8a-e6411eb7231b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"
Jul 29 19:41:16.783604 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.783588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f4bb8a-3175-40f1-aa8a-e6411eb7231b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r\" (UID: \"d1f4bb8a-3175-40f1-aa8a-e6411eb7231b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"
Jul 29 19:41:16.884912 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.884878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz\" (UID: \"94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"
Jul 29 19:41:16.885077 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.884940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d67k5\" (UniqueName: \"kubernetes.io/projected/d1f4bb8a-3175-40f1-aa8a-e6411eb7231b-kube-api-access-d67k5\") pod \"payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r\" (UID: \"d1f4bb8a-3175-40f1-aa8a-e6411eb7231b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"
Jul 29 19:41:16.885077 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.884980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqj9q\" (UniqueName: \"kubernetes.io/projected/94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc-kube-api-access-zqj9q\") pod \"payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz\" (UID: \"94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"
Jul 29 19:41:16.885077 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.885046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f4bb8a-3175-40f1-aa8a-e6411eb7231b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r\" (UID: \"d1f4bb8a-3175-40f1-aa8a-e6411eb7231b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"
Jul 29 19:41:16.885729 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.885708    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f4bb8a-3175-40f1-aa8a-e6411eb7231b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r\" (UID: \"d1f4bb8a-3175-40f1-aa8a-e6411eb7231b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"
Jul 29 19:41:16.897295 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.897266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d67k5\" (UniqueName: \"kubernetes.io/projected/d1f4bb8a-3175-40f1-aa8a-e6411eb7231b-kube-api-access-d67k5\") pod \"payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r\" (UID: \"d1f4bb8a-3175-40f1-aa8a-e6411eb7231b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"
Jul 29 19:41:16.985903 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.985871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz\" (UID: \"94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"
Jul 29 19:41:16.986063 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.985916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqj9q\" (UniqueName: \"kubernetes.io/projected/94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc-kube-api-access-zqj9q\") pod \"payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz\" (UID: \"94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"
Jul 29 19:41:16.986588 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.986567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz\" (UID: \"94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"
Jul 29 19:41:16.994633 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:16.994608    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqj9q\" (UniqueName: \"kubernetes.io/projected/94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc-kube-api-access-zqj9q\") pod \"payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz\" (UID: \"94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"
Jul 29 19:41:17.014689 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.014657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"
Jul 29 19:41:17.084054 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.084024    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"
Jul 29 19:41:17.188797 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.188763    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"]
Jul 29 19:41:17.190731 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:41:17.190708    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f4bb8a_3175_40f1_aa8a_e6411eb7231b.slice/crio-63edec7432f9ef1e5fb2512a702addd91ea4b9f1490d2f6318da9ff31702eecc WatchSource:0}: Error finding container 63edec7432f9ef1e5fb2512a702addd91ea4b9f1490d2f6318da9ff31702eecc: Status 404 returned error can't find the container with id 63edec7432f9ef1e5fb2512a702addd91ea4b9f1490d2f6318da9ff31702eecc
Jul 29 19:41:17.252341 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.252319    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"]
Jul 29 19:41:17.253826 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:41:17.253800    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b12a37_7eb8_4cfc_bfa6_e8daf98ae2bc.slice/crio-2a9245c4831595e34bf030db4318ef3a868e60409de57c60dc8a39caeaba3d00 WatchSource:0}: Error finding container 2a9245c4831595e34bf030db4318ef3a868e60409de57c60dc8a39caeaba3d00: Status 404 returned error can't find the container with id 2a9245c4831595e34bf030db4318ef3a868e60409de57c60dc8a39caeaba3d00
Jul 29 19:41:17.616103 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.616000    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs" event={"ID":"b5b643d1-1d19-4d5d-aff3-6822e9c4b096","Type":"ContainerStarted","Data":"a9e7ebc31db56ce8224fd7fac7354bdf04738936f3208dad4553e2a2c2584113"}
Jul 29 19:41:17.616340 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.616176    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:41:17.617736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.617702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r" event={"ID":"d1f4bb8a-3175-40f1-aa8a-e6411eb7231b","Type":"ContainerStarted","Data":"59569479bda22a65c9f5e47e35ef36623fec045c072f4c20e64efff6c6cc04b4"}
Jul 29 19:41:17.617918 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.617752    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"
Jul 29 19:41:17.617918 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.617771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r" event={"ID":"d1f4bb8a-3175-40f1-aa8a-e6411eb7231b","Type":"ContainerStarted","Data":"63edec7432f9ef1e5fb2512a702addd91ea4b9f1490d2f6318da9ff31702eecc"}
Jul 29 19:41:17.619275 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.619253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz" event={"ID":"94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc","Type":"ContainerStarted","Data":"290544c64e644ed055f03aa6a21acc6188885ec186fc6832750710851e99448e"}
Jul 29 19:41:17.619386 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.619287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz" event={"ID":"94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc","Type":"ContainerStarted","Data":"2a9245c4831595e34bf030db4318ef3a868e60409de57c60dc8a39caeaba3d00"}
Jul 29 19:41:17.619386 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.619381    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"
Jul 29 19:41:17.648160 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.648104    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs" podStartSLOduration=2.168629959 podStartE2EDuration="2.648087989s" totalImagesPullingTime="479.45803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:41:17.647236648 +0000 UTC m=+2443.946909168" watchObservedRunningTime="2026-07-29 19:41:17.648087989 +0000 UTC m=+2443.947760488"
Jul 29 19:41:17.681252 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.681173    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r" podStartSLOduration=1.681157402 podStartE2EDuration="1.681157402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:41:17.679439091 +0000 UTC m=+2443.979111590" watchObservedRunningTime="2026-07-29 19:41:17.681157402 +0000 UTC m=+2443.980829902"
Jul 29 19:41:17.721789 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:17.721731    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz" podStartSLOduration=1.7217131669999999 podStartE2EDuration="1.721713167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:41:17.720842992 +0000 UTC m=+2444.020515492" watchObservedRunningTime="2026-07-29 19:41:17.721713167 +0000 UTC m=+2444.021385721"
Jul 29 19:41:23.630990 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:23.630946    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:41:28.627629 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:28.627600    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"
Jul 29 19:41:28.628086 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:41:28.627946    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"
Jul 29 19:42:34.812459 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:34.812388    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"]
Jul 29 19:42:34.816045 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:34.816030    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:34.820165 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:34.820147    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b3bf1c\"/\"openshift-service-ca.crt\""
Jul 29 19:42:34.820959 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:34.820946    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b3bf1c\"/\"kube-root-ca.crt\""
Jul 29 19:42:34.821535 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:34.821516    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b3bf1c\"/\"default-dockercfg-tb7r9\""
Jul 29 19:42:34.821618 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:34.821552    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b3bf1c\"/\"test-model-b3bf1c-kserve-self-signed-certs\""
Jul 29 19:42:34.832519 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:34.832497    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"]
Jul 29 19:42:34.916094 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:34.916063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a69f7c-851c-45b9-8edf-a40b1494e497-tls-certs\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:34.916291 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:34.916102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxjxm\" (UniqueName: \"kubernetes.io/projected/b6a69f7c-851c-45b9-8edf-a40b1494e497-kube-api-access-lxjxm\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:34.916291 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:34.916156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-home\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:34.916291 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:34.916185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-tmp-dir\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:34.916291 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:34.916243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-model-cache\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:34.916291 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:34.916283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-dshm\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:35.017174 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:35.017146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxjxm\" (UniqueName: \"kubernetes.io/projected/b6a69f7c-851c-45b9-8edf-a40b1494e497-kube-api-access-lxjxm\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:35.017360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:35.017208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-home\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:35.017360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:35.017251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-tmp-dir\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:35.017470 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:35.017419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-model-cache\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:35.017544 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:35.017528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-dshm\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:35.017618 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:35.017573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a69f7c-851c-45b9-8edf-a40b1494e497-tls-certs\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:35.017666 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:35.017621    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-tmp-dir\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:35.017666 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:35.017638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-home\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:35.017764 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:35.017734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-model-cache\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:35.019732 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:35.019710    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-dshm\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:35.020067 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:35.020051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a69f7c-851c-45b9-8edf-a40b1494e497-tls-certs\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:35.026880 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:35.026859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxjxm\" (UniqueName: \"kubernetes.io/projected/b6a69f7c-851c-45b9-8edf-a40b1494e497-kube-api-access-lxjxm\") pod \"test-model-b3bf1c-kserve-75f6d77b75-kqqs8\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") " pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:35.124757 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:35.124680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:35.267578 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:35.267540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"]
Jul 29 19:42:35.268916 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:42:35.268892    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6a69f7c_851c_45b9_8edf_a40b1494e497.slice/crio-55da9fa7317cd9795d8fd0abe28ae8402bc6eb60d58f3da3380bd879f5dd7048 WatchSource:0}: Error finding container 55da9fa7317cd9795d8fd0abe28ae8402bc6eb60d58f3da3380bd879f5dd7048: Status 404 returned error can't find the container with id 55da9fa7317cd9795d8fd0abe28ae8402bc6eb60d58f3da3380bd879f5dd7048
Jul 29 19:42:36.024369 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:36.024335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8" event={"ID":"b6a69f7c-851c-45b9-8edf-a40b1494e497","Type":"ContainerStarted","Data":"2cfa78072c3cd51c6e5845197f2ad8cca28ab8032c71797689fb5798ae9d3c0c"}
Jul 29 19:42:36.024369 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:36.024373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8" event={"ID":"b6a69f7c-851c-45b9-8edf-a40b1494e497","Type":"ContainerStarted","Data":"55da9fa7317cd9795d8fd0abe28ae8402bc6eb60d58f3da3380bd879f5dd7048"}
Jul 29 19:42:36.042275 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:36.042232    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8" podStartSLOduration=2.042216918 podStartE2EDuration="2.042216918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:42:36.041235454 +0000 UTC m=+2522.340907955" watchObservedRunningTime="2026-07-29 19:42:36.042216918 +0000 UTC m=+2522.341889412"
Jul 29 19:42:45.125589 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:45.125542    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:45.125995 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:45.125609    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:45.137806 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:45.137779    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:42:46.085471 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:42:46.085445    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:43:21.547259 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:21.547220    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"]
Jul 29 19:43:32.220851 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.220810    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"]
Jul 29 19:43:32.221424 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.221095    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg" podUID="e43b01e8-6d2d-4de2-b7bc-3abbff6e4932" containerName="maas-api" containerID="cri-o://f4ef65f7ad310931eb6ed3412d93f7dd7cf710c83c24faafd32a383ff58b1a92" gracePeriod=30
Jul 29 19:43:32.274863 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.274826    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"]
Jul 29 19:43:32.275068 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.275027    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld" podUID="bc836c50-a140-48ec-a9b5-0fc0521798d6" containerName="payload-processing" containerID="cri-o://ed7192d6b0b309ab75bac032c703178ba8c257059422ac18458ed46ecdc91d11" gracePeriod=30
Jul 29 19:43:32.286063 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.286023    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"]
Jul 29 19:43:32.286366 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.286335    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff" podUID="8f153453-f8e9-4455-a6d3-8f936ecbdf79" containerName="payload-pre-processing" containerID="cri-o://15111d7e60a28006bd0ac61d1ea305626e935e5d1f1a5ace6526fe01335746af" gracePeriod=30
Jul 29 19:43:32.560079 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.560052    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:43:32.564476 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.564447    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"
Jul 29 19:43:32.584567 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.584537    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"
Jul 29 19:43:32.646134 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.646093    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-openshift-service-ca\") pod \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\" (UID: \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\") "
Jul 29 19:43:32.646339 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.646183    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f153453-f8e9-4455-a6d3-8f936ecbdf79-plugins-config-volume\") pod \"8f153453-f8e9-4455-a6d3-8f936ecbdf79\" (UID: \"8f153453-f8e9-4455-a6d3-8f936ecbdf79\") "
Jul 29 19:43:32.646339 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.646236    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87xs6\" (UniqueName: \"kubernetes.io/projected/bc836c50-a140-48ec-a9b5-0fc0521798d6-kube-api-access-87xs6\") pod \"bc836c50-a140-48ec-a9b5-0fc0521798d6\" (UID: \"bc836c50-a140-48ec-a9b5-0fc0521798d6\") "
Jul 29 19:43:32.646339 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.646269    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc836c50-a140-48ec-a9b5-0fc0521798d6-plugins-config-volume\") pod \"bc836c50-a140-48ec-a9b5-0fc0521798d6\" (UID: \"bc836c50-a140-48ec-a9b5-0fc0521798d6\") "
Jul 29 19:43:32.646511 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.646456    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnwj6\" (UniqueName: \"kubernetes.io/projected/8f153453-f8e9-4455-a6d3-8f936ecbdf79-kube-api-access-nnwj6\") pod \"8f153453-f8e9-4455-a6d3-8f936ecbdf79\" (UID: \"8f153453-f8e9-4455-a6d3-8f936ecbdf79\") "
Jul 29 19:43:32.646569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.646530    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-maas-api-tls\") pod \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\" (UID: \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\") "
Jul 29 19:43:32.646569 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.646559    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khbjq\" (UniqueName: \"kubernetes.io/projected/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-kube-api-access-khbjq\") pod \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\" (UID: \"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932\") "
Jul 29 19:43:32.646669 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.646647    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f153453-f8e9-4455-a6d3-8f936ecbdf79-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f153453-f8e9-4455-a6d3-8f936ecbdf79" (UID: "8f153453-f8e9-4455-a6d3-8f936ecbdf79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:43:32.646724 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.646686    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc836c50-a140-48ec-a9b5-0fc0521798d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc836c50-a140-48ec-a9b5-0fc0521798d6" (UID: "bc836c50-a140-48ec-a9b5-0fc0521798d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:43:32.646895 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.646873    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e43b01e8-6d2d-4de2-b7bc-3abbff6e4932" (UID: "e43b01e8-6d2d-4de2-b7bc-3abbff6e4932"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:43:32.646982 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.646914    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc836c50-a140-48ec-a9b5-0fc0521798d6-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:32.646982 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.646931    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f153453-f8e9-4455-a6d3-8f936ecbdf79-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:32.648651 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.648622    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc836c50-a140-48ec-a9b5-0fc0521798d6-kube-api-access-87xs6" (OuterVolumeSpecName: "kube-api-access-87xs6") pod "bc836c50-a140-48ec-a9b5-0fc0521798d6" (UID: "bc836c50-a140-48ec-a9b5-0fc0521798d6"). InnerVolumeSpecName "kube-api-access-87xs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:43:32.648777 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.648704    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-kube-api-access-khbjq" (OuterVolumeSpecName: "kube-api-access-khbjq") pod "e43b01e8-6d2d-4de2-b7bc-3abbff6e4932" (UID: "e43b01e8-6d2d-4de2-b7bc-3abbff6e4932"). InnerVolumeSpecName "kube-api-access-khbjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:43:32.648777 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.648750    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f153453-f8e9-4455-a6d3-8f936ecbdf79-kube-api-access-nnwj6" (OuterVolumeSpecName: "kube-api-access-nnwj6") pod "8f153453-f8e9-4455-a6d3-8f936ecbdf79" (UID: "8f153453-f8e9-4455-a6d3-8f936ecbdf79"). InnerVolumeSpecName "kube-api-access-nnwj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:43:32.649062 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.649047    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e43b01e8-6d2d-4de2-b7bc-3abbff6e4932" (UID: "e43b01e8-6d2d-4de2-b7bc-3abbff6e4932"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:43:32.747896 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.747798    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87xs6\" (UniqueName: \"kubernetes.io/projected/bc836c50-a140-48ec-a9b5-0fc0521798d6-kube-api-access-87xs6\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:32.747896 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.747835    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnwj6\" (UniqueName: \"kubernetes.io/projected/8f153453-f8e9-4455-a6d3-8f936ecbdf79-kube-api-access-nnwj6\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:32.747896 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.747846    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:32.747896 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.747855    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khbjq\" (UniqueName: \"kubernetes.io/projected/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-kube-api-access-khbjq\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:32.747896 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:32.747865    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:33.322274 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.322231    2579 generic.go:356] "Generic (PLEG): container finished" podID="8f153453-f8e9-4455-a6d3-8f936ecbdf79" containerID="15111d7e60a28006bd0ac61d1ea305626e935e5d1f1a5ace6526fe01335746af" exitCode=0
Jul 29 19:43:33.322755 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.322280    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"
Jul 29 19:43:33.322755 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.322320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff" event={"ID":"8f153453-f8e9-4455-a6d3-8f936ecbdf79","Type":"ContainerDied","Data":"15111d7e60a28006bd0ac61d1ea305626e935e5d1f1a5ace6526fe01335746af"}
Jul 29 19:43:33.322755 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.322373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff" event={"ID":"8f153453-f8e9-4455-a6d3-8f936ecbdf79","Type":"ContainerDied","Data":"1f0484db49511ab083ae117d13d27bd9c56080b940172d743988a1f41670eef0"}
Jul 29 19:43:33.322755 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.322407    2579 scope.go:117] "RemoveContainer" containerID="15111d7e60a28006bd0ac61d1ea305626e935e5d1f1a5ace6526fe01335746af"
Jul 29 19:43:33.323750 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.323643    2579 generic.go:356] "Generic (PLEG): container finished" podID="e43b01e8-6d2d-4de2-b7bc-3abbff6e4932" containerID="f4ef65f7ad310931eb6ed3412d93f7dd7cf710c83c24faafd32a383ff58b1a92" exitCode=0
Jul 29 19:43:33.323750 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.323675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg" event={"ID":"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932","Type":"ContainerDied","Data":"f4ef65f7ad310931eb6ed3412d93f7dd7cf710c83c24faafd32a383ff58b1a92"}
Jul 29 19:43:33.323750 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.323702    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"
Jul 29 19:43:33.323750 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.323726    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg" event={"ID":"e43b01e8-6d2d-4de2-b7bc-3abbff6e4932","Type":"ContainerDied","Data":"600f69636253d44e1920a243111195dc61032403480910663d3395bcbf43b925"}
Jul 29 19:43:33.325346 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.325322    2579 generic.go:356] "Generic (PLEG): container finished" podID="bc836c50-a140-48ec-a9b5-0fc0521798d6" containerID="ed7192d6b0b309ab75bac032c703178ba8c257059422ac18458ed46ecdc91d11" exitCode=0
Jul 29 19:43:33.325471 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.325363    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"
Jul 29 19:43:33.325471 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.325380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld" event={"ID":"bc836c50-a140-48ec-a9b5-0fc0521798d6","Type":"ContainerDied","Data":"ed7192d6b0b309ab75bac032c703178ba8c257059422ac18458ed46ecdc91d11"}
Jul 29 19:43:33.325471 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.325421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld" event={"ID":"bc836c50-a140-48ec-a9b5-0fc0521798d6","Type":"ContainerDied","Data":"8630db1a45adcb3d348f86f466a2ae372d612e79fb7502a0525cfbfcb8543084"}
Jul 29 19:43:33.334444 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.334415    2579 scope.go:117] "RemoveContainer" containerID="15111d7e60a28006bd0ac61d1ea305626e935e5d1f1a5ace6526fe01335746af"
Jul 29 19:43:33.339265 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:43:33.335116    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15111d7e60a28006bd0ac61d1ea305626e935e5d1f1a5ace6526fe01335746af\": container with ID starting with 15111d7e60a28006bd0ac61d1ea305626e935e5d1f1a5ace6526fe01335746af not found: ID does not exist" containerID="15111d7e60a28006bd0ac61d1ea305626e935e5d1f1a5ace6526fe01335746af"
Jul 29 19:43:33.339265 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.335160    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15111d7e60a28006bd0ac61d1ea305626e935e5d1f1a5ace6526fe01335746af"} err="failed to get container status \"15111d7e60a28006bd0ac61d1ea305626e935e5d1f1a5ace6526fe01335746af\": rpc error: code = NotFound desc = could not find container \"15111d7e60a28006bd0ac61d1ea305626e935e5d1f1a5ace6526fe01335746af\": container with ID starting with 15111d7e60a28006bd0ac61d1ea305626e935e5d1f1a5ace6526fe01335746af not found: ID does not exist"
Jul 29 19:43:33.339265 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.335224    2579 scope.go:117] "RemoveContainer" containerID="f4ef65f7ad310931eb6ed3412d93f7dd7cf710c83c24faafd32a383ff58b1a92"
Jul 29 19:43:33.349793 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.349769    2579 scope.go:117] "RemoveContainer" containerID="f4ef65f7ad310931eb6ed3412d93f7dd7cf710c83c24faafd32a383ff58b1a92"
Jul 29 19:43:33.350166 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:43:33.350139    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ef65f7ad310931eb6ed3412d93f7dd7cf710c83c24faafd32a383ff58b1a92\": container with ID starting with f4ef65f7ad310931eb6ed3412d93f7dd7cf710c83c24faafd32a383ff58b1a92 not found: ID does not exist" containerID="f4ef65f7ad310931eb6ed3412d93f7dd7cf710c83c24faafd32a383ff58b1a92"
Jul 29 19:43:33.350393 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.350176    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ef65f7ad310931eb6ed3412d93f7dd7cf710c83c24faafd32a383ff58b1a92"} err="failed to get container status \"f4ef65f7ad310931eb6ed3412d93f7dd7cf710c83c24faafd32a383ff58b1a92\": rpc error: code = NotFound desc = could not find container \"f4ef65f7ad310931eb6ed3412d93f7dd7cf710c83c24faafd32a383ff58b1a92\": container with ID starting with f4ef65f7ad310931eb6ed3412d93f7dd7cf710c83c24faafd32a383ff58b1a92 not found: ID does not exist"
Jul 29 19:43:33.350393 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.350208    2579 scope.go:117] "RemoveContainer" containerID="ed7192d6b0b309ab75bac032c703178ba8c257059422ac18458ed46ecdc91d11"
Jul 29 19:43:33.351274 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.351223    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"]
Jul 29 19:43:33.354410 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.354372    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-909676-59d895cc48-2dfff"]
Jul 29 19:43:33.360538 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.360485    2579 scope.go:117] "RemoveContainer" containerID="ed7192d6b0b309ab75bac032c703178ba8c257059422ac18458ed46ecdc91d11"
Jul 29 19:43:33.361028 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:43:33.360995    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7192d6b0b309ab75bac032c703178ba8c257059422ac18458ed46ecdc91d11\": container with ID starting with ed7192d6b0b309ab75bac032c703178ba8c257059422ac18458ed46ecdc91d11 not found: ID does not exist" containerID="ed7192d6b0b309ab75bac032c703178ba8c257059422ac18458ed46ecdc91d11"
Jul 29 19:43:33.361143 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.361039    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7192d6b0b309ab75bac032c703178ba8c257059422ac18458ed46ecdc91d11"} err="failed to get container status \"ed7192d6b0b309ab75bac032c703178ba8c257059422ac18458ed46ecdc91d11\": rpc error: code = NotFound desc = could not find container \"ed7192d6b0b309ab75bac032c703178ba8c257059422ac18458ed46ecdc91d11\": container with ID starting with ed7192d6b0b309ab75bac032c703178ba8c257059422ac18458ed46ecdc91d11 not found: ID does not exist"
Jul 29 19:43:33.367689 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.367660    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"]
Jul 29 19:43:33.373256 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.373228    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-909676-76d69d9bb9-g7prg"]
Jul 29 19:43:33.384317 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.384282    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"]
Jul 29 19:43:33.388605 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:33.388575    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-909676-565844c848-tnjld"]
Jul 29 19:43:34.196358 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:34.196324    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f153453-f8e9-4455-a6d3-8f936ecbdf79" path="/var/lib/kubelet/pods/8f153453-f8e9-4455-a6d3-8f936ecbdf79/volumes"
Jul 29 19:43:34.196719 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:34.196705    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc836c50-a140-48ec-a9b5-0fc0521798d6" path="/var/lib/kubelet/pods/bc836c50-a140-48ec-a9b5-0fc0521798d6/volumes"
Jul 29 19:43:34.197044 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:34.197033    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e43b01e8-6d2d-4de2-b7bc-3abbff6e4932" path="/var/lib/kubelet/pods/e43b01e8-6d2d-4de2-b7bc-3abbff6e4932/volumes"
Jul 29 19:43:36.566621 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.566578    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8" podUID="b6a69f7c-851c-45b9-8edf-a40b1494e497" containerName="main" containerID="cri-o://2cfa78072c3cd51c6e5845197f2ad8cca28ab8032c71797689fb5798ae9d3c0c" gracePeriod=45
Jul 29 19:43:36.822080 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.822010    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:43:36.987621 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.987579    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxjxm\" (UniqueName: \"kubernetes.io/projected/b6a69f7c-851c-45b9-8edf-a40b1494e497-kube-api-access-lxjxm\") pod \"b6a69f7c-851c-45b9-8edf-a40b1494e497\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") "
Jul 29 19:43:36.987825 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.987642    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-model-cache\") pod \"b6a69f7c-851c-45b9-8edf-a40b1494e497\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") "
Jul 29 19:43:36.987825 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.987773    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a69f7c-851c-45b9-8edf-a40b1494e497-tls-certs\") pod \"b6a69f7c-851c-45b9-8edf-a40b1494e497\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") "
Jul 29 19:43:36.987825 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.987818    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-tmp-dir\") pod \"b6a69f7c-851c-45b9-8edf-a40b1494e497\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") "
Jul 29 19:43:36.987997 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.987892    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-home\") pod \"b6a69f7c-851c-45b9-8edf-a40b1494e497\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") "
Jul 29 19:43:36.987997 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.987918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6a69f7c-851c-45b9-8edf-a40b1494e497" (UID: "b6a69f7c-851c-45b9-8edf-a40b1494e497"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:43:36.987997 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.987948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-dshm\") pod \"b6a69f7c-851c-45b9-8edf-a40b1494e497\" (UID: \"b6a69f7c-851c-45b9-8edf-a40b1494e497\") "
Jul 29 19:43:36.988136 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.988096    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6a69f7c-851c-45b9-8edf-a40b1494e497" (UID: "b6a69f7c-851c-45b9-8edf-a40b1494e497"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:43:36.988136 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.988110    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-home" (OuterVolumeSpecName: "home") pod "b6a69f7c-851c-45b9-8edf-a40b1494e497" (UID: "b6a69f7c-851c-45b9-8edf-a40b1494e497"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:43:36.988347 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.988321    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-tmp-dir\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:36.988347 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.988348    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-home\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:36.988507 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.988363    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-model-cache\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:36.990136 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.990102    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a69f7c-851c-45b9-8edf-a40b1494e497-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6a69f7c-851c-45b9-8edf-a40b1494e497" (UID: "b6a69f7c-851c-45b9-8edf-a40b1494e497"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:43:36.990275 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.990174    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a69f7c-851c-45b9-8edf-a40b1494e497-kube-api-access-lxjxm" (OuterVolumeSpecName: "kube-api-access-lxjxm") pod "b6a69f7c-851c-45b9-8edf-a40b1494e497" (UID: "b6a69f7c-851c-45b9-8edf-a40b1494e497"). InnerVolumeSpecName "kube-api-access-lxjxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:43:36.990275 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:36.990234    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-dshm" (OuterVolumeSpecName: "dshm") pod "b6a69f7c-851c-45b9-8edf-a40b1494e497" (UID: "b6a69f7c-851c-45b9-8edf-a40b1494e497"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:43:37.089432 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:37.089336    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a69f7c-851c-45b9-8edf-a40b1494e497-dshm\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:37.089432 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:37.089370    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxjxm\" (UniqueName: \"kubernetes.io/projected/b6a69f7c-851c-45b9-8edf-a40b1494e497-kube-api-access-lxjxm\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:37.089432 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:37.089384    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a69f7c-851c-45b9-8edf-a40b1494e497-tls-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:37.348808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:37.348721    2579 generic.go:356] "Generic (PLEG): container finished" podID="b6a69f7c-851c-45b9-8edf-a40b1494e497" containerID="2cfa78072c3cd51c6e5845197f2ad8cca28ab8032c71797689fb5798ae9d3c0c" exitCode=0
Jul 29 19:43:37.348808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:37.348770    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"
Jul 29 19:43:37.348808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:37.348771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8" event={"ID":"b6a69f7c-851c-45b9-8edf-a40b1494e497","Type":"ContainerDied","Data":"2cfa78072c3cd51c6e5845197f2ad8cca28ab8032c71797689fb5798ae9d3c0c"}
Jul 29 19:43:37.349153 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:37.348813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8" event={"ID":"b6a69f7c-851c-45b9-8edf-a40b1494e497","Type":"ContainerDied","Data":"55da9fa7317cd9795d8fd0abe28ae8402bc6eb60d58f3da3380bd879f5dd7048"}
Jul 29 19:43:37.349153 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:37.348837    2579 scope.go:117] "RemoveContainer" containerID="2cfa78072c3cd51c6e5845197f2ad8cca28ab8032c71797689fb5798ae9d3c0c"
Jul 29 19:43:37.358763 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:37.358741    2579 scope.go:117] "RemoveContainer" containerID="2cfa78072c3cd51c6e5845197f2ad8cca28ab8032c71797689fb5798ae9d3c0c"
Jul 29 19:43:37.359096 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:43:37.359076    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cfa78072c3cd51c6e5845197f2ad8cca28ab8032c71797689fb5798ae9d3c0c\": container with ID starting with 2cfa78072c3cd51c6e5845197f2ad8cca28ab8032c71797689fb5798ae9d3c0c not found: ID does not exist" containerID="2cfa78072c3cd51c6e5845197f2ad8cca28ab8032c71797689fb5798ae9d3c0c"
Jul 29 19:43:37.359167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:37.359109    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cfa78072c3cd51c6e5845197f2ad8cca28ab8032c71797689fb5798ae9d3c0c"} err="failed to get container status \"2cfa78072c3cd51c6e5845197f2ad8cca28ab8032c71797689fb5798ae9d3c0c\": rpc error: code = NotFound desc = could not find container \"2cfa78072c3cd51c6e5845197f2ad8cca28ab8032c71797689fb5798ae9d3c0c\": container with ID starting with 2cfa78072c3cd51c6e5845197f2ad8cca28ab8032c71797689fb5798ae9d3c0c not found: ID does not exist"
Jul 29 19:43:37.371337 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:37.371308    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"]
Jul 29 19:43:37.373082 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:37.373057    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b3bf1c/test-model-b3bf1c-kserve-75f6d77b75-kqqs8"]
Jul 29 19:43:38.196230 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:38.196176    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6a69f7c-851c-45b9-8edf-a40b1494e497" path="/var/lib/kubelet/pods/b6a69f7c-851c-45b9-8edf-a40b1494e497/volumes"
Jul 29 19:43:49.892951 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:49.892916    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"]
Jul 29 19:43:49.893493 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:49.893129    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs" podUID="b5b643d1-1d19-4d5d-aff3-6822e9c4b096" containerName="maas-api" containerID="cri-o://a9e7ebc31db56ce8224fd7fac7354bdf04738936f3208dad4553e2a2c2584113" gracePeriod=30
Jul 29 19:43:49.949504 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:49.949455    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"]
Jul 29 19:43:49.949994 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:49.949958    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r" podUID="d1f4bb8a-3175-40f1-aa8a-e6411eb7231b" containerName="payload-processing" containerID="cri-o://59569479bda22a65c9f5e47e35ef36623fec045c072f4c20e64efff6c6cc04b4" gracePeriod=30
Jul 29 19:43:49.955808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:49.955770    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"]
Jul 29 19:43:49.956279 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:49.955998    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz" podUID="94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc" containerName="payload-pre-processing" containerID="cri-o://290544c64e644ed055f03aa6a21acc6188885ec186fc6832750710851e99448e" gracePeriod=30
Jul 29 19:43:50.212796 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.212769    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:43:50.248748 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.248725    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"
Jul 29 19:43:50.254938 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.254911    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"
Jul 29 19:43:50.312300 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.312265    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-maas-api-tls\") pod \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\" (UID: \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\") "
Jul 29 19:43:50.312456 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.312379    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-openshift-service-ca\") pod \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\" (UID: \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\") "
Jul 29 19:43:50.312456 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.312407    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d67k5\" (UniqueName: \"kubernetes.io/projected/d1f4bb8a-3175-40f1-aa8a-e6411eb7231b-kube-api-access-d67k5\") pod \"d1f4bb8a-3175-40f1-aa8a-e6411eb7231b\" (UID: \"d1f4bb8a-3175-40f1-aa8a-e6411eb7231b\") "
Jul 29 19:43:50.312456 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.312431    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc-plugins-config-volume\") pod \"94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc\" (UID: \"94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc\") "
Jul 29 19:43:50.312592 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.312459    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29hsk\" (UniqueName: \"kubernetes.io/projected/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-kube-api-access-29hsk\") pod \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\" (UID: \"b5b643d1-1d19-4d5d-aff3-6822e9c4b096\") "
Jul 29 19:43:50.312592 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.312516    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqj9q\" (UniqueName: \"kubernetes.io/projected/94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc-kube-api-access-zqj9q\") pod \"94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc\" (UID: \"94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc\") "
Jul 29 19:43:50.312592 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.312545    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f4bb8a-3175-40f1-aa8a-e6411eb7231b-plugins-config-volume\") pod \"d1f4bb8a-3175-40f1-aa8a-e6411eb7231b\" (UID: \"d1f4bb8a-3175-40f1-aa8a-e6411eb7231b\") "
Jul 29 19:43:50.312889 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.312785    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b5b643d1-1d19-4d5d-aff3-6822e9c4b096" (UID: "b5b643d1-1d19-4d5d-aff3-6822e9c4b096"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:43:50.312889 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.312818    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc" (UID: "94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:43:50.313121 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.313086    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f4bb8a-3175-40f1-aa8a-e6411eb7231b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1f4bb8a-3175-40f1-aa8a-e6411eb7231b" (UID: "d1f4bb8a-3175-40f1-aa8a-e6411eb7231b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:43:50.314723 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.314697    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f4bb8a-3175-40f1-aa8a-e6411eb7231b-kube-api-access-d67k5" (OuterVolumeSpecName: "kube-api-access-d67k5") pod "d1f4bb8a-3175-40f1-aa8a-e6411eb7231b" (UID: "d1f4bb8a-3175-40f1-aa8a-e6411eb7231b"). InnerVolumeSpecName "kube-api-access-d67k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:43:50.314723 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.314717    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-kube-api-access-29hsk" (OuterVolumeSpecName: "kube-api-access-29hsk") pod "b5b643d1-1d19-4d5d-aff3-6822e9c4b096" (UID: "b5b643d1-1d19-4d5d-aff3-6822e9c4b096"). InnerVolumeSpecName "kube-api-access-29hsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:43:50.314977 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.314961    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b5b643d1-1d19-4d5d-aff3-6822e9c4b096" (UID: "b5b643d1-1d19-4d5d-aff3-6822e9c4b096"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:43:50.315043 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.315011    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc-kube-api-access-zqj9q" (OuterVolumeSpecName: "kube-api-access-zqj9q") pod "94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc" (UID: "94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc"). InnerVolumeSpecName "kube-api-access-zqj9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:43:50.413736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.413650    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:50.413736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.413676    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:50.413736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.413687    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d67k5\" (UniqueName: \"kubernetes.io/projected/d1f4bb8a-3175-40f1-aa8a-e6411eb7231b-kube-api-access-d67k5\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:50.413736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.413696    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:50.413736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.413704    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29hsk\" (UniqueName: \"kubernetes.io/projected/b5b643d1-1d19-4d5d-aff3-6822e9c4b096-kube-api-access-29hsk\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:50.413736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.413712    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqj9q\" (UniqueName: \"kubernetes.io/projected/94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc-kube-api-access-zqj9q\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:50.413736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.413720    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f4bb8a-3175-40f1-aa8a-e6411eb7231b-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:43:50.416850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.416822    2579 generic.go:356] "Generic (PLEG): container finished" podID="b5b643d1-1d19-4d5d-aff3-6822e9c4b096" containerID="a9e7ebc31db56ce8224fd7fac7354bdf04738936f3208dad4553e2a2c2584113" exitCode=0
Jul 29 19:43:50.417022 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.416865    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"
Jul 29 19:43:50.417022 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.416920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs" event={"ID":"b5b643d1-1d19-4d5d-aff3-6822e9c4b096","Type":"ContainerDied","Data":"a9e7ebc31db56ce8224fd7fac7354bdf04738936f3208dad4553e2a2c2584113"}
Jul 29 19:43:50.417022 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.416976    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs" event={"ID":"b5b643d1-1d19-4d5d-aff3-6822e9c4b096","Type":"ContainerDied","Data":"7d9b1f258ef9289a7dfe2b5ef2f0dd96c81a9285889354655b522bec65a41611"}
Jul 29 19:43:50.417022 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.417004    2579 scope.go:117] "RemoveContainer" containerID="a9e7ebc31db56ce8224fd7fac7354bdf04738936f3208dad4553e2a2c2584113"
Jul 29 19:43:50.418467 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.418430    2579 generic.go:356] "Generic (PLEG): container finished" podID="d1f4bb8a-3175-40f1-aa8a-e6411eb7231b" containerID="59569479bda22a65c9f5e47e35ef36623fec045c072f4c20e64efff6c6cc04b4" exitCode=0
Jul 29 19:43:50.418594 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.418471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r" event={"ID":"d1f4bb8a-3175-40f1-aa8a-e6411eb7231b","Type":"ContainerDied","Data":"59569479bda22a65c9f5e47e35ef36623fec045c072f4c20e64efff6c6cc04b4"}
Jul 29 19:43:50.418594 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.418532    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r" event={"ID":"d1f4bb8a-3175-40f1-aa8a-e6411eb7231b","Type":"ContainerDied","Data":"63edec7432f9ef1e5fb2512a702addd91ea4b9f1490d2f6318da9ff31702eecc"}
Jul 29 19:43:50.418594 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.418484    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"
Jul 29 19:43:50.419886 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.419864    2579 generic.go:356] "Generic (PLEG): container finished" podID="94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc" containerID="290544c64e644ed055f03aa6a21acc6188885ec186fc6832750710851e99448e" exitCode=0
Jul 29 19:43:50.420000 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.419924    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"
Jul 29 19:43:50.420000 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.419955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz" event={"ID":"94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc","Type":"ContainerDied","Data":"290544c64e644ed055f03aa6a21acc6188885ec186fc6832750710851e99448e"}
Jul 29 19:43:50.420000 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.419990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz" event={"ID":"94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc","Type":"ContainerDied","Data":"2a9245c4831595e34bf030db4318ef3a868e60409de57c60dc8a39caeaba3d00"}
Jul 29 19:43:50.428263 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.428129    2579 scope.go:117] "RemoveContainer" containerID="a9e7ebc31db56ce8224fd7fac7354bdf04738936f3208dad4553e2a2c2584113"
Jul 29 19:43:50.428606 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:43:50.428548    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9e7ebc31db56ce8224fd7fac7354bdf04738936f3208dad4553e2a2c2584113\": container with ID starting with a9e7ebc31db56ce8224fd7fac7354bdf04738936f3208dad4553e2a2c2584113 not found: ID does not exist" containerID="a9e7ebc31db56ce8224fd7fac7354bdf04738936f3208dad4553e2a2c2584113"
Jul 29 19:43:50.428606 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.428587    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e7ebc31db56ce8224fd7fac7354bdf04738936f3208dad4553e2a2c2584113"} err="failed to get container status \"a9e7ebc31db56ce8224fd7fac7354bdf04738936f3208dad4553e2a2c2584113\": rpc error: code = NotFound desc = could not find container \"a9e7ebc31db56ce8224fd7fac7354bdf04738936f3208dad4553e2a2c2584113\": container with ID starting with a9e7ebc31db56ce8224fd7fac7354bdf04738936f3208dad4553e2a2c2584113 not found: ID does not exist"
Jul 29 19:43:50.428789 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.428610    2579 scope.go:117] "RemoveContainer" containerID="59569479bda22a65c9f5e47e35ef36623fec045c072f4c20e64efff6c6cc04b4"
Jul 29 19:43:50.437621 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.437601    2579 scope.go:117] "RemoveContainer" containerID="59569479bda22a65c9f5e47e35ef36623fec045c072f4c20e64efff6c6cc04b4"
Jul 29 19:43:50.437904 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:43:50.437881    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59569479bda22a65c9f5e47e35ef36623fec045c072f4c20e64efff6c6cc04b4\": container with ID starting with 59569479bda22a65c9f5e47e35ef36623fec045c072f4c20e64efff6c6cc04b4 not found: ID does not exist" containerID="59569479bda22a65c9f5e47e35ef36623fec045c072f4c20e64efff6c6cc04b4"
Jul 29 19:43:50.437974 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.437912    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59569479bda22a65c9f5e47e35ef36623fec045c072f4c20e64efff6c6cc04b4"} err="failed to get container status \"59569479bda22a65c9f5e47e35ef36623fec045c072f4c20e64efff6c6cc04b4\": rpc error: code = NotFound desc = could not find container \"59569479bda22a65c9f5e47e35ef36623fec045c072f4c20e64efff6c6cc04b4\": container with ID starting with 59569479bda22a65c9f5e47e35ef36623fec045c072f4c20e64efff6c6cc04b4 not found: ID does not exist"
Jul 29 19:43:50.437974 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.437930    2579 scope.go:117] "RemoveContainer" containerID="290544c64e644ed055f03aa6a21acc6188885ec186fc6832750710851e99448e"
Jul 29 19:43:50.445628 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.445602    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"]
Jul 29 19:43:50.446400 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.446379    2579 scope.go:117] "RemoveContainer" containerID="290544c64e644ed055f03aa6a21acc6188885ec186fc6832750710851e99448e"
Jul 29 19:43:50.446748 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:43:50.446722    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290544c64e644ed055f03aa6a21acc6188885ec186fc6832750710851e99448e\": container with ID starting with 290544c64e644ed055f03aa6a21acc6188885ec186fc6832750710851e99448e not found: ID does not exist" containerID="290544c64e644ed055f03aa6a21acc6188885ec186fc6832750710851e99448e"
Jul 29 19:43:50.446822 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.446758    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290544c64e644ed055f03aa6a21acc6188885ec186fc6832750710851e99448e"} err="failed to get container status \"290544c64e644ed055f03aa6a21acc6188885ec186fc6832750710851e99448e\": rpc error: code = NotFound desc = could not find container \"290544c64e644ed055f03aa6a21acc6188885ec186fc6832750710851e99448e\": container with ID starting with 290544c64e644ed055f03aa6a21acc6188885ec186fc6832750710851e99448e not found: ID does not exist"
Jul 29 19:43:50.449553 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.449526    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bf1c-79cc9668bb-mvsqs"]
Jul 29 19:43:50.462129 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.462098    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"]
Jul 29 19:43:50.468082 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.468052    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3bf1c-67876f8d8b-8xp8r"]
Jul 29 19:43:50.480278 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.480243    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"]
Jul 29 19:43:50.481889 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:50.481869    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3bf1c-65d7b67d88-sm7fz"]
Jul 29 19:43:52.195674 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:52.195640    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc" path="/var/lib/kubelet/pods/94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc/volumes"
Jul 29 19:43:52.196043 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:52.196003    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5b643d1-1d19-4d5d-aff3-6822e9c4b096" path="/var/lib/kubelet/pods/b5b643d1-1d19-4d5d-aff3-6822e9c4b096/volumes"
Jul 29 19:43:52.196334 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:43:52.196321    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1f4bb8a-3175-40f1-aa8a-e6411eb7231b" path="/var/lib/kubelet/pods/d1f4bb8a-3175-40f1-aa8a-e6411eb7231b/volumes"
Jul 29 19:44:01.317527 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317446    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"]
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317803    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e43b01e8-6d2d-4de2-b7bc-3abbff6e4932" containerName="maas-api"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317813    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e43b01e8-6d2d-4de2-b7bc-3abbff6e4932" containerName="maas-api"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317824    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc836c50-a140-48ec-a9b5-0fc0521798d6" containerName="payload-processing"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317830    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc836c50-a140-48ec-a9b5-0fc0521798d6" containerName="payload-processing"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317836    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc" containerName="payload-pre-processing"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317842    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc" containerName="payload-pre-processing"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317848    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f4bb8a-3175-40f1-aa8a-e6411eb7231b" containerName="payload-processing"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317854    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f4bb8a-3175-40f1-aa8a-e6411eb7231b" containerName="payload-processing"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317860    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a69f7c-851c-45b9-8edf-a40b1494e497" containerName="main"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317864    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a69f7c-851c-45b9-8edf-a40b1494e497" containerName="main"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317874    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f153453-f8e9-4455-a6d3-8f936ecbdf79" containerName="payload-pre-processing"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317879    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f153453-f8e9-4455-a6d3-8f936ecbdf79" containerName="payload-pre-processing"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317885    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b643d1-1d19-4d5d-aff3-6822e9c4b096" containerName="maas-api"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317889    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b643d1-1d19-4d5d-aff3-6822e9c4b096" containerName="maas-api"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317942    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b643d1-1d19-4d5d-aff3-6822e9c4b096" containerName="maas-api"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317950    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f4bb8a-3175-40f1-aa8a-e6411eb7231b" containerName="payload-processing"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317956    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f153453-f8e9-4455-a6d3-8f936ecbdf79" containerName="payload-pre-processing"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317964    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b12a37-7eb8-4cfc-bfa6-e8daf98ae2bc" containerName="payload-pre-processing"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317972    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e43b01e8-6d2d-4de2-b7bc-3abbff6e4932" containerName="maas-api"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317976    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a69f7c-851c-45b9-8edf-a40b1494e497" containerName="main"
Jul 29 19:44:01.317985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.317983    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc836c50-a140-48ec-a9b5-0fc0521798d6" containerName="payload-processing"
Jul 29 19:44:01.322581 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.322561    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:01.327961 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.327820    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9e7a39-serving-cert\""
Jul 29 19:44:01.327961 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.327944    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:44:01.328234 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.328213    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:44:01.329423 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.329261    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5bzmf\""
Jul 29 19:44:01.343750 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.343722    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"]
Jul 29 19:44:01.410245 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.410183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-maas-api-tls\") pod \"maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn\" (UID: \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:01.410409 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.410280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdrkq\" (UniqueName: \"kubernetes.io/projected/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-kube-api-access-wdrkq\") pod \"maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn\" (UID: \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:01.410409 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.410307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-openshift-service-ca\") pod \"maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn\" (UID: \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:01.511042 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.511003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdrkq\" (UniqueName: \"kubernetes.io/projected/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-kube-api-access-wdrkq\") pod \"maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn\" (UID: \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:01.511042 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.511046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-openshift-service-ca\") pod \"maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn\" (UID: \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:01.511267 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.511102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-maas-api-tls\") pod \"maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn\" (UID: \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:01.511726 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.511706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-openshift-service-ca\") pod \"maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn\" (UID: \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:01.513608 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.513588    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-maas-api-tls\") pod \"maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn\" (UID: \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:01.531775 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.531744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdrkq\" (UniqueName: \"kubernetes.io/projected/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-kube-api-access-wdrkq\") pod \"maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn\" (UID: \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:01.632815 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.632781    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:01.770059 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.770024    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"]
Jul 29 19:44:01.773411 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:01.773385    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:44:02.076430 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.076346    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"]
Jul 29 19:44:02.081251 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.081178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"
Jul 29 19:44:02.085060 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.085040    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9e7a39\""
Jul 29 19:44:02.096212 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.096172    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"]
Jul 29 19:44:02.100031 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.100013    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"]
Jul 29 19:44:02.100101 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.100094    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"
Jul 29 19:44:02.104145 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.104125    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9e7a39-dockercfg-vbznt\""
Jul 29 19:44:02.115922 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.115894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889adde8-f66a-4d02-9da4-50d9b0d19036-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q\" (UID: \"889adde8-f66a-4d02-9da4-50d9b0d19036\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"
Jul 29 19:44:02.116096 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.116050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgc5m\" (UniqueName: \"kubernetes.io/projected/889adde8-f66a-4d02-9da4-50d9b0d19036-kube-api-access-mgc5m\") pod \"payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q\" (UID: \"889adde8-f66a-4d02-9da4-50d9b0d19036\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"
Jul 29 19:44:02.120993 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.120962    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"]
Jul 29 19:44:02.217113 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.217078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2xw8\" (UniqueName: \"kubernetes.io/projected/3a533498-0d2b-4cfa-a910-98cc3c7a46b3-kube-api-access-v2xw8\") pod \"payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn\" (UID: \"3a533498-0d2b-4cfa-a910-98cc3c7a46b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"
Jul 29 19:44:02.217295 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.217135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889adde8-f66a-4d02-9da4-50d9b0d19036-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q\" (UID: \"889adde8-f66a-4d02-9da4-50d9b0d19036\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"
Jul 29 19:44:02.217295 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.217157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a533498-0d2b-4cfa-a910-98cc3c7a46b3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn\" (UID: \"3a533498-0d2b-4cfa-a910-98cc3c7a46b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"
Jul 29 19:44:02.217295 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.217176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgc5m\" (UniqueName: \"kubernetes.io/projected/889adde8-f66a-4d02-9da4-50d9b0d19036-kube-api-access-mgc5m\") pod \"payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q\" (UID: \"889adde8-f66a-4d02-9da4-50d9b0d19036\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"
Jul 29 19:44:02.218034 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.218011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889adde8-f66a-4d02-9da4-50d9b0d19036-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q\" (UID: \"889adde8-f66a-4d02-9da4-50d9b0d19036\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"
Jul 29 19:44:02.227334 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.227306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgc5m\" (UniqueName: \"kubernetes.io/projected/889adde8-f66a-4d02-9da4-50d9b0d19036-kube-api-access-mgc5m\") pod \"payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q\" (UID: \"889adde8-f66a-4d02-9da4-50d9b0d19036\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"
Jul 29 19:44:02.318214 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.318156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2xw8\" (UniqueName: \"kubernetes.io/projected/3a533498-0d2b-4cfa-a910-98cc3c7a46b3-kube-api-access-v2xw8\") pod \"payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn\" (UID: \"3a533498-0d2b-4cfa-a910-98cc3c7a46b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"
Jul 29 19:44:02.318661 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.318268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a533498-0d2b-4cfa-a910-98cc3c7a46b3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn\" (UID: \"3a533498-0d2b-4cfa-a910-98cc3c7a46b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"
Jul 29 19:44:02.319090 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.319067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a533498-0d2b-4cfa-a910-98cc3c7a46b3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn\" (UID: \"3a533498-0d2b-4cfa-a910-98cc3c7a46b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"
Jul 29 19:44:02.327915 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.327854    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2xw8\" (UniqueName: \"kubernetes.io/projected/3a533498-0d2b-4cfa-a910-98cc3c7a46b3-kube-api-access-v2xw8\") pod \"payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn\" (UID: \"3a533498-0d2b-4cfa-a910-98cc3c7a46b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"
Jul 29 19:44:02.392252 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.392231    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"
Jul 29 19:44:02.410093 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.410069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"
Jul 29 19:44:02.485230 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.485012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn" event={"ID":"c8c50d77-cd7f-41f4-bf07-7a3d7084e923","Type":"ContainerStarted","Data":"fafe9fced589a38b6aed108f3bf035a9feebac20ed968faaad199387af50e9af"}
Jul 29 19:44:02.485230 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.485082    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn" event={"ID":"c8c50d77-cd7f-41f4-bf07-7a3d7084e923","Type":"ContainerStarted","Data":"fa768d49c62047c18867a15cec3a66e6bfcd656e5b0bd2c1586ebc7462d4ccb8"}
Jul 29 19:44:02.485230 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.485117    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:02.506952 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.506417    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn" podStartSLOduration=0.890743443 podStartE2EDuration="1.506401843s" totalImagesPullingTime="615.6584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:44:02.506084498 +0000 UTC m=+2608.805756997" watchObservedRunningTime="2026-07-29 19:44:02.506401843 +0000 UTC m=+2608.806074341"
Jul 29 19:44:02.547762 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.547725    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"]
Jul 29 19:44:02.550257 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:44:02.550226    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889adde8_f66a_4d02_9da4_50d9b0d19036.slice/crio-b3f258ec4128dbf3dbc2b0cc3a9c42cbd2acccb39d7252fd3577f1467621c8ac WatchSource:0}: Error finding container b3f258ec4128dbf3dbc2b0cc3a9c42cbd2acccb39d7252fd3577f1467621c8ac: Status 404 returned error can't find the container with id b3f258ec4128dbf3dbc2b0cc3a9c42cbd2acccb39d7252fd3577f1467621c8ac
Jul 29 19:44:02.578112 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:02.577991    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"]
Jul 29 19:44:02.580372 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:44:02.580340    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a533498_0d2b_4cfa_a910_98cc3c7a46b3.slice/crio-4b6d90c9389befa5ba527466aea384a0e13a2df95e9a5b1caaad57255c1c0ad6 WatchSource:0}: Error finding container 4b6d90c9389befa5ba527466aea384a0e13a2df95e9a5b1caaad57255c1c0ad6: Status 404 returned error can't find the container with id 4b6d90c9389befa5ba527466aea384a0e13a2df95e9a5b1caaad57255c1c0ad6
Jul 29 19:44:03.491702 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:03.491667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn" event={"ID":"3a533498-0d2b-4cfa-a910-98cc3c7a46b3","Type":"ContainerStarted","Data":"01d6592f5993d8c674a97d50ff87cf951cf99c01b02bad830173e33a507d3ec7"}
Jul 29 19:44:03.492330 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:03.491725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn" event={"ID":"3a533498-0d2b-4cfa-a910-98cc3c7a46b3","Type":"ContainerStarted","Data":"4b6d90c9389befa5ba527466aea384a0e13a2df95e9a5b1caaad57255c1c0ad6"}
Jul 29 19:44:03.492330 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:03.491771    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"
Jul 29 19:44:03.493574 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:03.493547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q" event={"ID":"889adde8-f66a-4d02-9da4-50d9b0d19036","Type":"ContainerStarted","Data":"592cbfde1a085406f5326c002662a46f08e2113e9160887de7755aeacc4893c2"}
Jul 29 19:44:03.493694 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:03.493597    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q" event={"ID":"889adde8-f66a-4d02-9da4-50d9b0d19036","Type":"ContainerStarted","Data":"b3f258ec4128dbf3dbc2b0cc3a9c42cbd2acccb39d7252fd3577f1467621c8ac"}
Jul 29 19:44:03.493750 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:03.493691    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"
Jul 29 19:44:03.512944 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:03.512884    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn" podStartSLOduration=1.512865318 podStartE2EDuration="1.512865318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:44:03.511775119 +0000 UTC m=+2609.811447645" watchObservedRunningTime="2026-07-29 19:44:03.512865318 +0000 UTC m=+2609.812537796"
Jul 29 19:44:03.529567 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:03.529513    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q" podStartSLOduration=1.529496014 podStartE2EDuration="1.529496014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:44:03.528393837 +0000 UTC m=+2609.828066337" watchObservedRunningTime="2026-07-29 19:44:03.529496014 +0000 UTC m=+2609.829168513"
Jul 29 19:44:08.499641 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:08.499612    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:14.210886 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.210849    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"]
Jul 29 19:44:14.221081 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.220178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:44:14.223764 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.223735    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f9937e-serving-cert\""
Jul 29 19:44:14.226357 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.226328    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"]
Jul 29 19:44:14.333589 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.333547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qvq\" (UniqueName: \"kubernetes.io/projected/2036ee81-4cd2-44d9-bea1-8d989d99ab75-kube-api-access-p5qvq\") pod \"maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g\" (UID: \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:44:14.333788 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.333596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2036ee81-4cd2-44d9-bea1-8d989d99ab75-maas-api-tls\") pod \"maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g\" (UID: \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:44:14.333856 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.333777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2036ee81-4cd2-44d9-bea1-8d989d99ab75-openshift-service-ca\") pod \"maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g\" (UID: \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:44:14.435187 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.435147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2036ee81-4cd2-44d9-bea1-8d989d99ab75-openshift-service-ca\") pod \"maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g\" (UID: \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:44:14.435420 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.435264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qvq\" (UniqueName: \"kubernetes.io/projected/2036ee81-4cd2-44d9-bea1-8d989d99ab75-kube-api-access-p5qvq\") pod \"maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g\" (UID: \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:44:14.435420 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.435293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2036ee81-4cd2-44d9-bea1-8d989d99ab75-maas-api-tls\") pod \"maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g\" (UID: \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:44:14.435546 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:44:14.435459    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-serving-cert: secret "maas-api-e2e-api-b-f9937e-serving-cert" not found
Jul 29 19:44:14.435546 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:44:14.435536    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2036ee81-4cd2-44d9-bea1-8d989d99ab75-maas-api-tls podName:2036ee81-4cd2-44d9-bea1-8d989d99ab75 nodeName:}" failed. No retries permitted until 2026-07-29 19:44:14.935512699 +0000 UTC m=+2621.235185181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2036ee81-4cd2-44d9-bea1-8d989d99ab75-maas-api-tls") pod "maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g" (UID: "2036ee81-4cd2-44d9-bea1-8d989d99ab75") : secret "maas-api-e2e-api-b-f9937e-serving-cert" not found
Jul 29 19:44:14.435943 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.435918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2036ee81-4cd2-44d9-bea1-8d989d99ab75-openshift-service-ca\") pod \"maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g\" (UID: \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:44:14.446654 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.446632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qvq\" (UniqueName: \"kubernetes.io/projected/2036ee81-4cd2-44d9-bea1-8d989d99ab75-kube-api-access-p5qvq\") pod \"maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g\" (UID: \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:44:14.501226 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.501114    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"
Jul 29 19:44:14.501226 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.501221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"
Jul 29 19:44:14.938771 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.938728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2036ee81-4cd2-44d9-bea1-8d989d99ab75-maas-api-tls\") pod \"maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g\" (UID: \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:44:14.941416 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.941397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2036ee81-4cd2-44d9-bea1-8d989d99ab75-maas-api-tls\") pod \"maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g\" (UID: \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:44:14.993927 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.993893    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"]
Jul 29 19:44:14.997644 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:14.997629    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"
Jul 29 19:44:15.001298 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.001278    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f9937e-dockercfg-tzxwv\""
Jul 29 19:44:15.001642 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.001624    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f9937e\""
Jul 29 19:44:15.006236 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.006186    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"]
Jul 29 19:44:15.084470 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.084431    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"]
Jul 29 19:44:15.088631 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.088608    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"
Jul 29 19:44:15.102208 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.102163    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"]
Jul 29 19:44:15.137493 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.137462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:44:15.140687 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.140334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c436af-599b-42ec-a2ba-337ded1bf6d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc\" (UID: \"84c436af-599b-42ec-a2ba-337ded1bf6d1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"
Jul 29 19:44:15.140687 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.140417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77pg5\" (UniqueName: \"kubernetes.io/projected/84c436af-599b-42ec-a2ba-337ded1bf6d1-kube-api-access-77pg5\") pod \"payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc\" (UID: \"84c436af-599b-42ec-a2ba-337ded1bf6d1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"
Jul 29 19:44:15.241802 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.241767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c436af-599b-42ec-a2ba-337ded1bf6d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc\" (UID: \"84c436af-599b-42ec-a2ba-337ded1bf6d1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"
Jul 29 19:44:15.242309 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.241822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w6bn\" (UniqueName: \"kubernetes.io/projected/88c0df2a-e9c4-4a9e-ad78-fffb34d749d3-kube-api-access-4w6bn\") pod \"payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c\" (UID: \"88c0df2a-e9c4-4a9e-ad78-fffb34d749d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"
Jul 29 19:44:15.242309 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.241864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88c0df2a-e9c4-4a9e-ad78-fffb34d749d3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c\" (UID: \"88c0df2a-e9c4-4a9e-ad78-fffb34d749d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"
Jul 29 19:44:15.242309 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.241897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77pg5\" (UniqueName: \"kubernetes.io/projected/84c436af-599b-42ec-a2ba-337ded1bf6d1-kube-api-access-77pg5\") pod \"payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc\" (UID: \"84c436af-599b-42ec-a2ba-337ded1bf6d1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"
Jul 29 19:44:15.242585 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.242567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c436af-599b-42ec-a2ba-337ded1bf6d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc\" (UID: \"84c436af-599b-42ec-a2ba-337ded1bf6d1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"
Jul 29 19:44:15.251451 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.251408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77pg5\" (UniqueName: \"kubernetes.io/projected/84c436af-599b-42ec-a2ba-337ded1bf6d1-kube-api-access-77pg5\") pod \"payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc\" (UID: \"84c436af-599b-42ec-a2ba-337ded1bf6d1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"
Jul 29 19:44:15.309305 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.309274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"
Jul 29 19:44:15.342620 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.342582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w6bn\" (UniqueName: \"kubernetes.io/projected/88c0df2a-e9c4-4a9e-ad78-fffb34d749d3-kube-api-access-4w6bn\") pod \"payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c\" (UID: \"88c0df2a-e9c4-4a9e-ad78-fffb34d749d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"
Jul 29 19:44:15.342782 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.342653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88c0df2a-e9c4-4a9e-ad78-fffb34d749d3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c\" (UID: \"88c0df2a-e9c4-4a9e-ad78-fffb34d749d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"
Jul 29 19:44:15.343621 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.343592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88c0df2a-e9c4-4a9e-ad78-fffb34d749d3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c\" (UID: \"88c0df2a-e9c4-4a9e-ad78-fffb34d749d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"
Jul 29 19:44:15.351607 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.351578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w6bn\" (UniqueName: \"kubernetes.io/projected/88c0df2a-e9c4-4a9e-ad78-fffb34d749d3-kube-api-access-4w6bn\") pod \"payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c\" (UID: \"88c0df2a-e9c4-4a9e-ad78-fffb34d749d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"
Jul 29 19:44:15.399570 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.399524    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"
Jul 29 19:44:15.444821 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.444771    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"]
Jul 29 19:44:15.447989 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:44:15.447942    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c436af_599b_42ec_a2ba_337ded1bf6d1.slice/crio-5e7fcf32b54cd75961f7f12d4dd26c275d5048805a95f182c99a2a491b1c41cf WatchSource:0}: Error finding container 5e7fcf32b54cd75961f7f12d4dd26c275d5048805a95f182c99a2a491b1c41cf: Status 404 returned error can't find the container with id 5e7fcf32b54cd75961f7f12d4dd26c275d5048805a95f182c99a2a491b1c41cf
Jul 29 19:44:15.497338 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.496453    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"]
Jul 29 19:44:15.498841 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:44:15.498800    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2036ee81_4cd2_44d9_bea1_8d989d99ab75.slice/crio-83a738822776eef5a05cdf0e3ba16d969bf57a609671e019f7ba365df6d8afc5 WatchSource:0}: Error finding container 83a738822776eef5a05cdf0e3ba16d969bf57a609671e019f7ba365df6d8afc5: Status 404 returned error can't find the container with id 83a738822776eef5a05cdf0e3ba16d969bf57a609671e019f7ba365df6d8afc5
Jul 29 19:44:15.552390 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.552368    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"]
Jul 29 19:44:15.554523 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:44:15.554496    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c0df2a_e9c4_4a9e_ad78_fffb34d749d3.slice/crio-131a7d5cdd1eeea35c003243986a49c1177a3f22de68fa36dfff28bf2fbc2fe3 WatchSource:0}: Error finding container 131a7d5cdd1eeea35c003243986a49c1177a3f22de68fa36dfff28bf2fbc2fe3: Status 404 returned error can't find the container with id 131a7d5cdd1eeea35c003243986a49c1177a3f22de68fa36dfff28bf2fbc2fe3
Jul 29 19:44:15.557637 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.557607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g" event={"ID":"2036ee81-4cd2-44d9-bea1-8d989d99ab75","Type":"ContainerStarted","Data":"83a738822776eef5a05cdf0e3ba16d969bf57a609671e019f7ba365df6d8afc5"}
Jul 29 19:44:15.559099 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.559077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc" event={"ID":"84c436af-599b-42ec-a2ba-337ded1bf6d1","Type":"ContainerStarted","Data":"af4f671d9fb5ed1f4245f113a36d9083f1d14b1bb18b595414c83ddb1d435557"}
Jul 29 19:44:15.559180 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.559128    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc" event={"ID":"84c436af-599b-42ec-a2ba-337ded1bf6d1","Type":"ContainerStarted","Data":"5e7fcf32b54cd75961f7f12d4dd26c275d5048805a95f182c99a2a491b1c41cf"}
Jul 29 19:44:15.560676 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.560654    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"
Jul 29 19:44:15.585631 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:15.585578    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc" podStartSLOduration=1.585561152 podStartE2EDuration="1.585561152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:44:15.583345728 +0000 UTC m=+2621.883018283" watchObservedRunningTime="2026-07-29 19:44:15.585561152 +0000 UTC m=+2621.885233651"
Jul 29 19:44:16.565665 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:16.565622    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c" event={"ID":"88c0df2a-e9c4-4a9e-ad78-fffb34d749d3","Type":"ContainerStarted","Data":"d0684afca8b8db1a3ff3ee103339b6eca3532fe54a0e68ea5017030a498095d0"}
Jul 29 19:44:16.566158 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:16.565680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c" event={"ID":"88c0df2a-e9c4-4a9e-ad78-fffb34d749d3","Type":"ContainerStarted","Data":"131a7d5cdd1eeea35c003243986a49c1177a3f22de68fa36dfff28bf2fbc2fe3"}
Jul 29 19:44:16.566158 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:16.565715    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"
Jul 29 19:44:16.567160 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:16.567137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g" event={"ID":"2036ee81-4cd2-44d9-bea1-8d989d99ab75","Type":"ContainerStarted","Data":"4af4cb432076065ec5066e89c9d748b2b6764505f9666d9dc71b9071efb93cd9"}
Jul 29 19:44:16.606344 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:16.606280    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c" podStartSLOduration=1.6062565439999998 podStartE2EDuration="1.606256544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:44:16.602413428 +0000 UTC m=+2622.902085960" watchObservedRunningTime="2026-07-29 19:44:16.606256544 +0000 UTC m=+2622.905929053"
Jul 29 19:44:16.629888 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:16.629825    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g" podStartSLOduration=2.296227429 podStartE2EDuration="2.629802764s" totalImagesPullingTime="333.575335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:44:16.628886531 +0000 UTC m=+2622.928559047" watchObservedRunningTime="2026-07-29 19:44:16.629802764 +0000 UTC m=+2622.929475267"
Jul 29 19:44:17.573271 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:17.573237    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:44:23.583675 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:23.583649    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:44:26.569756 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:26.569723    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"
Jul 29 19:44:27.576021 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:27.575991    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"
Jul 29 19:44:42.230845 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.230808    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"]
Jul 29 19:44:42.231311 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.231012    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn" podUID="c8c50d77-cd7f-41f4-bf07-7a3d7084e923" containerName="maas-api" containerID="cri-o://fafe9fced589a38b6aed108f3bf035a9feebac20ed968faaad199387af50e9af" gracePeriod=30
Jul 29 19:44:42.283166 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.283125    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"]
Jul 29 19:44:42.283449 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.283415    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn" podUID="3a533498-0d2b-4cfa-a910-98cc3c7a46b3" containerName="payload-processing" containerID="cri-o://01d6592f5993d8c674a97d50ff87cf951cf99c01b02bad830173e33a507d3ec7" gracePeriod=30
Jul 29 19:44:42.298335 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.298296    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"]
Jul 29 19:44:42.298603 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.298573    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q" podUID="889adde8-f66a-4d02-9da4-50d9b0d19036" containerName="payload-pre-processing" containerID="cri-o://592cbfde1a085406f5326c002662a46f08e2113e9160887de7755aeacc4893c2" gracePeriod=30
Jul 29 19:44:42.560683 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.560651    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:42.568654 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.568598    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"
Jul 29 19:44:42.595234 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.594672    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdrkq\" (UniqueName: \"kubernetes.io/projected/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-kube-api-access-wdrkq\") pod \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\" (UID: \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\") "
Jul 29 19:44:42.595234 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.594757    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-openshift-service-ca\") pod \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\" (UID: \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\") "
Jul 29 19:44:42.595234 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.594800    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-maas-api-tls\") pod \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\" (UID: \"c8c50d77-cd7f-41f4-bf07-7a3d7084e923\") "
Jul 29 19:44:42.595234 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.594820    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2xw8\" (UniqueName: \"kubernetes.io/projected/3a533498-0d2b-4cfa-a910-98cc3c7a46b3-kube-api-access-v2xw8\") pod \"3a533498-0d2b-4cfa-a910-98cc3c7a46b3\" (UID: \"3a533498-0d2b-4cfa-a910-98cc3c7a46b3\") "
Jul 29 19:44:42.595234 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.594856    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a533498-0d2b-4cfa-a910-98cc3c7a46b3-plugins-config-volume\") pod \"3a533498-0d2b-4cfa-a910-98cc3c7a46b3\" (UID: \"3a533498-0d2b-4cfa-a910-98cc3c7a46b3\") "
Jul 29 19:44:42.595629 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.595377    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8c50d77-cd7f-41f4-bf07-7a3d7084e923" (UID: "c8c50d77-cd7f-41f4-bf07-7a3d7084e923"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:44:42.595629 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.595462    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a533498-0d2b-4cfa-a910-98cc3c7a46b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a533498-0d2b-4cfa-a910-98cc3c7a46b3" (UID: "3a533498-0d2b-4cfa-a910-98cc3c7a46b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:44:42.597162 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.597135    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-kube-api-access-wdrkq" (OuterVolumeSpecName: "kube-api-access-wdrkq") pod "c8c50d77-cd7f-41f4-bf07-7a3d7084e923" (UID: "c8c50d77-cd7f-41f4-bf07-7a3d7084e923"). InnerVolumeSpecName "kube-api-access-wdrkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:44:42.597343 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.597176    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"
Jul 29 19:44:42.597618 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.597594    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8c50d77-cd7f-41f4-bf07-7a3d7084e923" (UID: "c8c50d77-cd7f-41f4-bf07-7a3d7084e923"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:44:42.597713 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.597689    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a533498-0d2b-4cfa-a910-98cc3c7a46b3-kube-api-access-v2xw8" (OuterVolumeSpecName: "kube-api-access-v2xw8") pod "3a533498-0d2b-4cfa-a910-98cc3c7a46b3" (UID: "3a533498-0d2b-4cfa-a910-98cc3c7a46b3"). InnerVolumeSpecName "kube-api-access-v2xw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:44:42.695835 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.695795    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgc5m\" (UniqueName: \"kubernetes.io/projected/889adde8-f66a-4d02-9da4-50d9b0d19036-kube-api-access-mgc5m\") pod \"889adde8-f66a-4d02-9da4-50d9b0d19036\" (UID: \"889adde8-f66a-4d02-9da4-50d9b0d19036\") "
Jul 29 19:44:42.696009 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.695859    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889adde8-f66a-4d02-9da4-50d9b0d19036-plugins-config-volume\") pod \"889adde8-f66a-4d02-9da4-50d9b0d19036\" (UID: \"889adde8-f66a-4d02-9da4-50d9b0d19036\") "
Jul 29 19:44:42.696090 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.696078    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:44:42.696128 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.696093    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2xw8\" (UniqueName: \"kubernetes.io/projected/3a533498-0d2b-4cfa-a910-98cc3c7a46b3-kube-api-access-v2xw8\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:44:42.696128 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.696111    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a533498-0d2b-4cfa-a910-98cc3c7a46b3-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:44:42.696128 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.696124    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdrkq\" (UniqueName: \"kubernetes.io/projected/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-kube-api-access-wdrkq\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:44:42.696278 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.696134    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c50d77-cd7f-41f4-bf07-7a3d7084e923-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:44:42.696315 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.696273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889adde8-f66a-4d02-9da4-50d9b0d19036-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "889adde8-f66a-4d02-9da4-50d9b0d19036" (UID: "889adde8-f66a-4d02-9da4-50d9b0d19036"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:44:42.698267 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.698244    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889adde8-f66a-4d02-9da4-50d9b0d19036-kube-api-access-mgc5m" (OuterVolumeSpecName: "kube-api-access-mgc5m") pod "889adde8-f66a-4d02-9da4-50d9b0d19036" (UID: "889adde8-f66a-4d02-9da4-50d9b0d19036"). InnerVolumeSpecName "kube-api-access-mgc5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:44:42.712630 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.712592    2579 generic.go:356] "Generic (PLEG): container finished" podID="889adde8-f66a-4d02-9da4-50d9b0d19036" containerID="592cbfde1a085406f5326c002662a46f08e2113e9160887de7755aeacc4893c2" exitCode=0
Jul 29 19:44:42.712804 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.712642    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"
Jul 29 19:44:42.712804 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.712680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q" event={"ID":"889adde8-f66a-4d02-9da4-50d9b0d19036","Type":"ContainerDied","Data":"592cbfde1a085406f5326c002662a46f08e2113e9160887de7755aeacc4893c2"}
Jul 29 19:44:42.712804 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.712761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q" event={"ID":"889adde8-f66a-4d02-9da4-50d9b0d19036","Type":"ContainerDied","Data":"b3f258ec4128dbf3dbc2b0cc3a9c42cbd2acccb39d7252fd3577f1467621c8ac"}
Jul 29 19:44:42.712804 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.712791    2579 scope.go:117] "RemoveContainer" containerID="592cbfde1a085406f5326c002662a46f08e2113e9160887de7755aeacc4893c2"
Jul 29 19:44:42.714292 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.714249    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8c50d77-cd7f-41f4-bf07-7a3d7084e923" containerID="fafe9fced589a38b6aed108f3bf035a9feebac20ed968faaad199387af50e9af" exitCode=0
Jul 29 19:44:42.714427 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.714316    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"
Jul 29 19:44:42.714498 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.714329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn" event={"ID":"c8c50d77-cd7f-41f4-bf07-7a3d7084e923","Type":"ContainerDied","Data":"fafe9fced589a38b6aed108f3bf035a9feebac20ed968faaad199387af50e9af"}
Jul 29 19:44:42.714566 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.714488    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn" event={"ID":"c8c50d77-cd7f-41f4-bf07-7a3d7084e923","Type":"ContainerDied","Data":"fa768d49c62047c18867a15cec3a66e6bfcd656e5b0bd2c1586ebc7462d4ccb8"}
Jul 29 19:44:42.715969 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.715934    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a533498-0d2b-4cfa-a910-98cc3c7a46b3" containerID="01d6592f5993d8c674a97d50ff87cf951cf99c01b02bad830173e33a507d3ec7" exitCode=0
Jul 29 19:44:42.716082 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.715986    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"
Jul 29 19:44:42.716082 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.716018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn" event={"ID":"3a533498-0d2b-4cfa-a910-98cc3c7a46b3","Type":"ContainerDied","Data":"01d6592f5993d8c674a97d50ff87cf951cf99c01b02bad830173e33a507d3ec7"}
Jul 29 19:44:42.716082 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.716071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn" event={"ID":"3a533498-0d2b-4cfa-a910-98cc3c7a46b3","Type":"ContainerDied","Data":"4b6d90c9389befa5ba527466aea384a0e13a2df95e9a5b1caaad57255c1c0ad6"}
Jul 29 19:44:42.729086 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.729067    2579 scope.go:117] "RemoveContainer" containerID="592cbfde1a085406f5326c002662a46f08e2113e9160887de7755aeacc4893c2"
Jul 29 19:44:42.729518 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:44:42.729488    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592cbfde1a085406f5326c002662a46f08e2113e9160887de7755aeacc4893c2\": container with ID starting with 592cbfde1a085406f5326c002662a46f08e2113e9160887de7755aeacc4893c2 not found: ID does not exist" containerID="592cbfde1a085406f5326c002662a46f08e2113e9160887de7755aeacc4893c2"
Jul 29 19:44:42.729638 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.729533    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592cbfde1a085406f5326c002662a46f08e2113e9160887de7755aeacc4893c2"} err="failed to get container status \"592cbfde1a085406f5326c002662a46f08e2113e9160887de7755aeacc4893c2\": rpc error: code = NotFound desc = could not find container \"592cbfde1a085406f5326c002662a46f08e2113e9160887de7755aeacc4893c2\": container with ID starting with 592cbfde1a085406f5326c002662a46f08e2113e9160887de7755aeacc4893c2 not found: ID does not exist"
Jul 29 19:44:42.729638 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.729561    2579 scope.go:117] "RemoveContainer" containerID="fafe9fced589a38b6aed108f3bf035a9feebac20ed968faaad199387af50e9af"
Jul 29 19:44:42.739069 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.739047    2579 scope.go:117] "RemoveContainer" containerID="fafe9fced589a38b6aed108f3bf035a9feebac20ed968faaad199387af50e9af"
Jul 29 19:44:42.739386 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:44:42.739361    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fafe9fced589a38b6aed108f3bf035a9feebac20ed968faaad199387af50e9af\": container with ID starting with fafe9fced589a38b6aed108f3bf035a9feebac20ed968faaad199387af50e9af not found: ID does not exist" containerID="fafe9fced589a38b6aed108f3bf035a9feebac20ed968faaad199387af50e9af"
Jul 29 19:44:42.739501 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.739393    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fafe9fced589a38b6aed108f3bf035a9feebac20ed968faaad199387af50e9af"} err="failed to get container status \"fafe9fced589a38b6aed108f3bf035a9feebac20ed968faaad199387af50e9af\": rpc error: code = NotFound desc = could not find container \"fafe9fced589a38b6aed108f3bf035a9feebac20ed968faaad199387af50e9af\": container with ID starting with fafe9fced589a38b6aed108f3bf035a9feebac20ed968faaad199387af50e9af not found: ID does not exist"
Jul 29 19:44:42.739501 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.739419    2579 scope.go:117] "RemoveContainer" containerID="01d6592f5993d8c674a97d50ff87cf951cf99c01b02bad830173e33a507d3ec7"
Jul 29 19:44:42.748187 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.748160    2579 scope.go:117] "RemoveContainer" containerID="01d6592f5993d8c674a97d50ff87cf951cf99c01b02bad830173e33a507d3ec7"
Jul 29 19:44:42.748583 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:44:42.748548    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d6592f5993d8c674a97d50ff87cf951cf99c01b02bad830173e33a507d3ec7\": container with ID starting with 01d6592f5993d8c674a97d50ff87cf951cf99c01b02bad830173e33a507d3ec7 not found: ID does not exist" containerID="01d6592f5993d8c674a97d50ff87cf951cf99c01b02bad830173e33a507d3ec7"
Jul 29 19:44:42.748705 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.748595    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d6592f5993d8c674a97d50ff87cf951cf99c01b02bad830173e33a507d3ec7"} err="failed to get container status \"01d6592f5993d8c674a97d50ff87cf951cf99c01b02bad830173e33a507d3ec7\": rpc error: code = NotFound desc = could not find container \"01d6592f5993d8c674a97d50ff87cf951cf99c01b02bad830173e33a507d3ec7\": container with ID starting with 01d6592f5993d8c674a97d50ff87cf951cf99c01b02bad830173e33a507d3ec7 not found: ID does not exist"
Jul 29 19:44:42.751241 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.751212    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"]
Jul 29 19:44:42.757469 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.757440    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9e7a39-595dd78588-mjd4q"]
Jul 29 19:44:42.771319 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.771280    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"]
Jul 29 19:44:42.774279 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.774250    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9e7a39-5cb476d64f-mqkrn"]
Jul 29 19:44:42.786339 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.786305    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"]
Jul 29 19:44:42.790378 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.790345    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9e7a39-776f998c9b-mb2gn"]
Jul 29 19:44:42.797111 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.797076    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889adde8-f66a-4d02-9da4-50d9b0d19036-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:44:42.797111 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:42.797110    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgc5m\" (UniqueName: \"kubernetes.io/projected/889adde8-f66a-4d02-9da4-50d9b0d19036-kube-api-access-mgc5m\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:44:44.196346 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:44.196313    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a533498-0d2b-4cfa-a910-98cc3c7a46b3" path="/var/lib/kubelet/pods/3a533498-0d2b-4cfa-a910-98cc3c7a46b3/volumes"
Jul 29 19:44:44.196740 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:44.196671    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="889adde8-f66a-4d02-9da4-50d9b0d19036" path="/var/lib/kubelet/pods/889adde8-f66a-4d02-9da4-50d9b0d19036/volumes"
Jul 29 19:44:44.197011 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:44:44.196997    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8c50d77-cd7f-41f4-bf07-7a3d7084e923" path="/var/lib/kubelet/pods/c8c50d77-cd7f-41f4-bf07-7a3d7084e923/volumes"
Jul 29 19:45:00.142409 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.142377    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"]
Jul 29 19:45:00.142820 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.142595    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g" podUID="2036ee81-4cd2-44d9-bea1-8d989d99ab75" containerName="maas-api" containerID="cri-o://4af4cb432076065ec5066e89c9d748b2b6764505f9666d9dc71b9071efb93cd9" gracePeriod=30
Jul 29 19:45:00.241792 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.241757    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"]
Jul 29 19:45:00.241999 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.241977    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c" podUID="88c0df2a-e9c4-4a9e-ad78-fffb34d749d3" containerName="payload-processing" containerID="cri-o://d0684afca8b8db1a3ff3ee103339b6eca3532fe54a0e68ea5017030a498095d0" gracePeriod=30
Jul 29 19:45:00.249559 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.249520    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"]
Jul 29 19:45:00.249817 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.249788    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc" podUID="84c436af-599b-42ec-a2ba-337ded1bf6d1" containerName="payload-pre-processing" containerID="cri-o://af4f671d9fb5ed1f4245f113a36d9083f1d14b1bb18b595414c83ddb1d435557" gracePeriod=30
Jul 29 19:45:00.411572 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.411546    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:45:00.461602 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.461572    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2036ee81-4cd2-44d9-bea1-8d989d99ab75-maas-api-tls\") pod \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\" (UID: \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\") "
Jul 29 19:45:00.461747 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.461677    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2036ee81-4cd2-44d9-bea1-8d989d99ab75-openshift-service-ca\") pod \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\" (UID: \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\") "
Jul 29 19:45:00.461794 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.461753    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5qvq\" (UniqueName: \"kubernetes.io/projected/2036ee81-4cd2-44d9-bea1-8d989d99ab75-kube-api-access-p5qvq\") pod \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\" (UID: \"2036ee81-4cd2-44d9-bea1-8d989d99ab75\") "
Jul 29 19:45:00.462219 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.462159    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2036ee81-4cd2-44d9-bea1-8d989d99ab75-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2036ee81-4cd2-44d9-bea1-8d989d99ab75" (UID: "2036ee81-4cd2-44d9-bea1-8d989d99ab75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:45:00.464472 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.464449    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2036ee81-4cd2-44d9-bea1-8d989d99ab75-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2036ee81-4cd2-44d9-bea1-8d989d99ab75" (UID: "2036ee81-4cd2-44d9-bea1-8d989d99ab75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:45:00.464626 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.464599    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2036ee81-4cd2-44d9-bea1-8d989d99ab75-kube-api-access-p5qvq" (OuterVolumeSpecName: "kube-api-access-p5qvq") pod "2036ee81-4cd2-44d9-bea1-8d989d99ab75" (UID: "2036ee81-4cd2-44d9-bea1-8d989d99ab75"). InnerVolumeSpecName "kube-api-access-p5qvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:45:00.503837 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.503805    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"
Jul 29 19:45:00.512358 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.512330    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"
Jul 29 19:45:00.562300 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.562271    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88c0df2a-e9c4-4a9e-ad78-fffb34d749d3-plugins-config-volume\") pod \"88c0df2a-e9c4-4a9e-ad78-fffb34d749d3\" (UID: \"88c0df2a-e9c4-4a9e-ad78-fffb34d749d3\") "
Jul 29 19:45:00.562466 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.562355    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77pg5\" (UniqueName: \"kubernetes.io/projected/84c436af-599b-42ec-a2ba-337ded1bf6d1-kube-api-access-77pg5\") pod \"84c436af-599b-42ec-a2ba-337ded1bf6d1\" (UID: \"84c436af-599b-42ec-a2ba-337ded1bf6d1\") "
Jul 29 19:45:00.562466 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.562382    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w6bn\" (UniqueName: \"kubernetes.io/projected/88c0df2a-e9c4-4a9e-ad78-fffb34d749d3-kube-api-access-4w6bn\") pod \"88c0df2a-e9c4-4a9e-ad78-fffb34d749d3\" (UID: \"88c0df2a-e9c4-4a9e-ad78-fffb34d749d3\") "
Jul 29 19:45:00.562608 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.562506    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c436af-599b-42ec-a2ba-337ded1bf6d1-plugins-config-volume\") pod \"84c436af-599b-42ec-a2ba-337ded1bf6d1\" (UID: \"84c436af-599b-42ec-a2ba-337ded1bf6d1\") "
Jul 29 19:45:00.562768 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.562723    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c0df2a-e9c4-4a9e-ad78-fffb34d749d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "88c0df2a-e9c4-4a9e-ad78-fffb34d749d3" (UID: "88c0df2a-e9c4-4a9e-ad78-fffb34d749d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:45:00.562986 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.562963    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c436af-599b-42ec-a2ba-337ded1bf6d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84c436af-599b-42ec-a2ba-337ded1bf6d1" (UID: "84c436af-599b-42ec-a2ba-337ded1bf6d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:45:00.563089 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.562966    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2036ee81-4cd2-44d9-bea1-8d989d99ab75-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:45:00.563089 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.563012    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5qvq\" (UniqueName: \"kubernetes.io/projected/2036ee81-4cd2-44d9-bea1-8d989d99ab75-kube-api-access-p5qvq\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:45:00.563089 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.563026    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88c0df2a-e9c4-4a9e-ad78-fffb34d749d3-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:45:00.563089 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.563040    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2036ee81-4cd2-44d9-bea1-8d989d99ab75-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:45:00.564675 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.564652    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c0df2a-e9c4-4a9e-ad78-fffb34d749d3-kube-api-access-4w6bn" (OuterVolumeSpecName: "kube-api-access-4w6bn") pod "88c0df2a-e9c4-4a9e-ad78-fffb34d749d3" (UID: "88c0df2a-e9c4-4a9e-ad78-fffb34d749d3"). InnerVolumeSpecName "kube-api-access-4w6bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:45:00.565056 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.565041    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c436af-599b-42ec-a2ba-337ded1bf6d1-kube-api-access-77pg5" (OuterVolumeSpecName: "kube-api-access-77pg5") pod "84c436af-599b-42ec-a2ba-337ded1bf6d1" (UID: "84c436af-599b-42ec-a2ba-337ded1bf6d1"). InnerVolumeSpecName "kube-api-access-77pg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:45:00.664107 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.664014    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77pg5\" (UniqueName: \"kubernetes.io/projected/84c436af-599b-42ec-a2ba-337ded1bf6d1-kube-api-access-77pg5\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:45:00.664107 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.664044    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w6bn\" (UniqueName: \"kubernetes.io/projected/88c0df2a-e9c4-4a9e-ad78-fffb34d749d3-kube-api-access-4w6bn\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:45:00.664107 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.664053    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c436af-599b-42ec-a2ba-337ded1bf6d1-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:45:00.809644 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.809607    2579 generic.go:356] "Generic (PLEG): container finished" podID="2036ee81-4cd2-44d9-bea1-8d989d99ab75" containerID="4af4cb432076065ec5066e89c9d748b2b6764505f9666d9dc71b9071efb93cd9" exitCode=0
Jul 29 19:45:00.809846 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.809656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g" event={"ID":"2036ee81-4cd2-44d9-bea1-8d989d99ab75","Type":"ContainerDied","Data":"4af4cb432076065ec5066e89c9d748b2b6764505f9666d9dc71b9071efb93cd9"}
Jul 29 19:45:00.809846 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.809738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g" event={"ID":"2036ee81-4cd2-44d9-bea1-8d989d99ab75","Type":"ContainerDied","Data":"83a738822776eef5a05cdf0e3ba16d969bf57a609671e019f7ba365df6d8afc5"}
Jul 29 19:45:00.809846 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.809671    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"
Jul 29 19:45:00.809846 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.809766    2579 scope.go:117] "RemoveContainer" containerID="4af4cb432076065ec5066e89c9d748b2b6764505f9666d9dc71b9071efb93cd9"
Jul 29 19:45:00.811151 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.811121    2579 generic.go:356] "Generic (PLEG): container finished" podID="88c0df2a-e9c4-4a9e-ad78-fffb34d749d3" containerID="d0684afca8b8db1a3ff3ee103339b6eca3532fe54a0e68ea5017030a498095d0" exitCode=0
Jul 29 19:45:00.811296 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.811174    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"
Jul 29 19:45:00.811296 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.811222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c" event={"ID":"88c0df2a-e9c4-4a9e-ad78-fffb34d749d3","Type":"ContainerDied","Data":"d0684afca8b8db1a3ff3ee103339b6eca3532fe54a0e68ea5017030a498095d0"}
Jul 29 19:45:00.811296 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.811274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c" event={"ID":"88c0df2a-e9c4-4a9e-ad78-fffb34d749d3","Type":"ContainerDied","Data":"131a7d5cdd1eeea35c003243986a49c1177a3f22de68fa36dfff28bf2fbc2fe3"}
Jul 29 19:45:00.812888 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.812862    2579 generic.go:356] "Generic (PLEG): container finished" podID="84c436af-599b-42ec-a2ba-337ded1bf6d1" containerID="af4f671d9fb5ed1f4245f113a36d9083f1d14b1bb18b595414c83ddb1d435557" exitCode=0
Jul 29 19:45:00.813005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.812899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc" event={"ID":"84c436af-599b-42ec-a2ba-337ded1bf6d1","Type":"ContainerDied","Data":"af4f671d9fb5ed1f4245f113a36d9083f1d14b1bb18b595414c83ddb1d435557"}
Jul 29 19:45:00.813005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.812949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc" event={"ID":"84c436af-599b-42ec-a2ba-337ded1bf6d1","Type":"ContainerDied","Data":"5e7fcf32b54cd75961f7f12d4dd26c275d5048805a95f182c99a2a491b1c41cf"}
Jul 29 19:45:00.813219 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.813173    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"
Jul 29 19:45:00.820892 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.820874    2579 scope.go:117] "RemoveContainer" containerID="4af4cb432076065ec5066e89c9d748b2b6764505f9666d9dc71b9071efb93cd9"
Jul 29 19:45:00.821240 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:45:00.821212    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af4cb432076065ec5066e89c9d748b2b6764505f9666d9dc71b9071efb93cd9\": container with ID starting with 4af4cb432076065ec5066e89c9d748b2b6764505f9666d9dc71b9071efb93cd9 not found: ID does not exist" containerID="4af4cb432076065ec5066e89c9d748b2b6764505f9666d9dc71b9071efb93cd9"
Jul 29 19:45:00.821451 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.821249    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af4cb432076065ec5066e89c9d748b2b6764505f9666d9dc71b9071efb93cd9"} err="failed to get container status \"4af4cb432076065ec5066e89c9d748b2b6764505f9666d9dc71b9071efb93cd9\": rpc error: code = NotFound desc = could not find container \"4af4cb432076065ec5066e89c9d748b2b6764505f9666d9dc71b9071efb93cd9\": container with ID starting with 4af4cb432076065ec5066e89c9d748b2b6764505f9666d9dc71b9071efb93cd9 not found: ID does not exist"
Jul 29 19:45:00.821451 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.821268    2579 scope.go:117] "RemoveContainer" containerID="d0684afca8b8db1a3ff3ee103339b6eca3532fe54a0e68ea5017030a498095d0"
Jul 29 19:45:00.830640 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.830619    2579 scope.go:117] "RemoveContainer" containerID="d0684afca8b8db1a3ff3ee103339b6eca3532fe54a0e68ea5017030a498095d0"
Jul 29 19:45:00.830973 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:45:00.830951    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0684afca8b8db1a3ff3ee103339b6eca3532fe54a0e68ea5017030a498095d0\": container with ID starting with d0684afca8b8db1a3ff3ee103339b6eca3532fe54a0e68ea5017030a498095d0 not found: ID does not exist" containerID="d0684afca8b8db1a3ff3ee103339b6eca3532fe54a0e68ea5017030a498095d0"
Jul 29 19:45:00.831027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.830981    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0684afca8b8db1a3ff3ee103339b6eca3532fe54a0e68ea5017030a498095d0"} err="failed to get container status \"d0684afca8b8db1a3ff3ee103339b6eca3532fe54a0e68ea5017030a498095d0\": rpc error: code = NotFound desc = could not find container \"d0684afca8b8db1a3ff3ee103339b6eca3532fe54a0e68ea5017030a498095d0\": container with ID starting with d0684afca8b8db1a3ff3ee103339b6eca3532fe54a0e68ea5017030a498095d0 not found: ID does not exist"
Jul 29 19:45:00.831027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.831000    2579 scope.go:117] "RemoveContainer" containerID="af4f671d9fb5ed1f4245f113a36d9083f1d14b1bb18b595414c83ddb1d435557"
Jul 29 19:45:00.839489 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.839465    2579 scope.go:117] "RemoveContainer" containerID="af4f671d9fb5ed1f4245f113a36d9083f1d14b1bb18b595414c83ddb1d435557"
Jul 29 19:45:00.839795 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:45:00.839779    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af4f671d9fb5ed1f4245f113a36d9083f1d14b1bb18b595414c83ddb1d435557\": container with ID starting with af4f671d9fb5ed1f4245f113a36d9083f1d14b1bb18b595414c83ddb1d435557 not found: ID does not exist" containerID="af4f671d9fb5ed1f4245f113a36d9083f1d14b1bb18b595414c83ddb1d435557"
Jul 29 19:45:00.839841 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.839802    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af4f671d9fb5ed1f4245f113a36d9083f1d14b1bb18b595414c83ddb1d435557"} err="failed to get container status \"af4f671d9fb5ed1f4245f113a36d9083f1d14b1bb18b595414c83ddb1d435557\": rpc error: code = NotFound desc = could not find container \"af4f671d9fb5ed1f4245f113a36d9083f1d14b1bb18b595414c83ddb1d435557\": container with ID starting with af4f671d9fb5ed1f4245f113a36d9083f1d14b1bb18b595414c83ddb1d435557 not found: ID does not exist"
Jul 29 19:45:00.850892 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.850863    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"]
Jul 29 19:45:00.852598 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.852570    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9937e-5d6c5b68df-tvp8g"]
Jul 29 19:45:00.868697 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.868663    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"]
Jul 29 19:45:00.874888 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.874859    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9937e-c974c4596-pdnfc"]
Jul 29 19:45:00.891565 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.891532    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"]
Jul 29 19:45:00.894794 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:00.894767    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9937e-7cff6f8956-fsp8c"]
Jul 29 19:45:02.195491 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:02.195462    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2036ee81-4cd2-44d9-bea1-8d989d99ab75" path="/var/lib/kubelet/pods/2036ee81-4cd2-44d9-bea1-8d989d99ab75/volumes"
Jul 29 19:45:02.195899 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:02.195798    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84c436af-599b-42ec-a2ba-337ded1bf6d1" path="/var/lib/kubelet/pods/84c436af-599b-42ec-a2ba-337ded1bf6d1/volumes"
Jul 29 19:45:02.196097 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:02.196085    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c0df2a-e9c4-4a9e-ad78-fffb34d749d3" path="/var/lib/kubelet/pods/88c0df2a-e9c4-4a9e-ad78-fffb34d749d3/volumes"
Jul 29 19:45:11.292819 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.292778    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"]
Jul 29 19:45:11.293575 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293553    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="889adde8-f66a-4d02-9da4-50d9b0d19036" containerName="payload-pre-processing"
Jul 29 19:45:11.293670 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293581    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="889adde8-f66a-4d02-9da4-50d9b0d19036" containerName="payload-pre-processing"
Jul 29 19:45:11.293670 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293603    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c436af-599b-42ec-a2ba-337ded1bf6d1" containerName="payload-pre-processing"
Jul 29 19:45:11.293670 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293611    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c436af-599b-42ec-a2ba-337ded1bf6d1" containerName="payload-pre-processing"
Jul 29 19:45:11.293670 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293625    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a533498-0d2b-4cfa-a910-98cc3c7a46b3" containerName="payload-processing"
Jul 29 19:45:11.293670 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293634    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a533498-0d2b-4cfa-a910-98cc3c7a46b3" containerName="payload-processing"
Jul 29 19:45:11.293670 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293646    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2036ee81-4cd2-44d9-bea1-8d989d99ab75" containerName="maas-api"
Jul 29 19:45:11.293670 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293654    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2036ee81-4cd2-44d9-bea1-8d989d99ab75" containerName="maas-api"
Jul 29 19:45:11.294017 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293682    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c50d77-cd7f-41f4-bf07-7a3d7084e923" containerName="maas-api"
Jul 29 19:45:11.294017 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293690    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c50d77-cd7f-41f4-bf07-7a3d7084e923" containerName="maas-api"
Jul 29 19:45:11.294017 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293700    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c0df2a-e9c4-4a9e-ad78-fffb34d749d3" containerName="payload-processing"
Jul 29 19:45:11.294017 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293708    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c0df2a-e9c4-4a9e-ad78-fffb34d749d3" containerName="payload-processing"
Jul 29 19:45:11.294017 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293815    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="889adde8-f66a-4d02-9da4-50d9b0d19036" containerName="payload-pre-processing"
Jul 29 19:45:11.294017 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293826    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a533498-0d2b-4cfa-a910-98cc3c7a46b3" containerName="payload-processing"
Jul 29 19:45:11.294017 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293834    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2036ee81-4cd2-44d9-bea1-8d989d99ab75" containerName="maas-api"
Jul 29 19:45:11.294017 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293845    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c50d77-cd7f-41f4-bf07-7a3d7084e923" containerName="maas-api"
Jul 29 19:45:11.294017 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293853    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="84c436af-599b-42ec-a2ba-337ded1bf6d1" containerName="payload-pre-processing"
Jul 29 19:45:11.294017 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.293861    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c0df2a-e9c4-4a9e-ad78-fffb34d749d3" containerName="payload-processing"
Jul 29 19:45:11.299564 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.299539    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:45:11.310473 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.310446    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-8e241a-serving-cert\""
Jul 29 19:45:11.310612 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.310514    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:45:11.322313 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.322280    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5bzmf\""
Jul 29 19:45:11.323440 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.323409    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:45:11.325427 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.325401    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"]
Jul 29 19:45:11.468502 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.468470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v47hv\" (UniqueName: \"kubernetes.io/projected/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-kube-api-access-v47hv\") pod \"maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw\" (UID: \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:45:11.468714 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.468510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-maas-api-tls\") pod \"maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw\" (UID: \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:45:11.468714 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.468543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw\" (UID: \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:45:11.569110 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.569023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47hv\" (UniqueName: \"kubernetes.io/projected/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-kube-api-access-v47hv\") pod \"maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw\" (UID: \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:45:11.569110 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.569066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-maas-api-tls\") pod \"maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw\" (UID: \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:45:11.569110 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.569097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw\" (UID: \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:45:11.569868 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.569790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw\" (UID: \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:45:11.571793 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.571763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-maas-api-tls\") pod \"maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw\" (UID: \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:45:11.583811 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.583779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v47hv\" (UniqueName: \"kubernetes.io/projected/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-kube-api-access-v47hv\") pod \"maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw\" (UID: \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:45:11.609970 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.609925    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:45:11.776278 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.776243    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"]
Jul 29 19:45:11.777784 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:45:11.777755    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e264efc_fae3_459d_9a39_cf4c9f8fdae8.slice/crio-d3e0a293510c29853caf4d15903d702750e66d6824667cfeb4230f4c4ce310df WatchSource:0}: Error finding container d3e0a293510c29853caf4d15903d702750e66d6824667cfeb4230f4c4ce310df: Status 404 returned error can't find the container with id d3e0a293510c29853caf4d15903d702750e66d6824667cfeb4230f4c4ce310df
Jul 29 19:45:11.875129 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:11.875033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw" event={"ID":"1e264efc-fae3-459d-9a39-cf4c9f8fdae8","Type":"ContainerStarted","Data":"d3e0a293510c29853caf4d15903d702750e66d6824667cfeb4230f4c4ce310df"}
Jul 29 19:45:12.157028 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.156985    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"]
Jul 29 19:45:12.161650 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.161635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"
Jul 29 19:45:12.164865 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.164841    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-8e241a\""
Jul 29 19:45:12.168698 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.168674    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"]
Jul 29 19:45:12.274786 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.274746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e029d7af-1acb-4676-af97-c16fc92d8c20-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg\" (UID: \"e029d7af-1acb-4676-af97-c16fc92d8c20\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"
Jul 29 19:45:12.274956 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.274930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cfdx\" (UniqueName: \"kubernetes.io/projected/e029d7af-1acb-4676-af97-c16fc92d8c20-kube-api-access-8cfdx\") pod \"payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg\" (UID: \"e029d7af-1acb-4676-af97-c16fc92d8c20\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"
Jul 29 19:45:12.311762 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.311731    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"]
Jul 29 19:45:12.315234 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.315215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"
Jul 29 19:45:12.318027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.318008    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-8e241a-dockercfg-6ns8c\""
Jul 29 19:45:12.328979 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.328954    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"]
Jul 29 19:45:12.375674 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.375639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cfdx\" (UniqueName: \"kubernetes.io/projected/e029d7af-1acb-4676-af97-c16fc92d8c20-kube-api-access-8cfdx\") pod \"payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg\" (UID: \"e029d7af-1acb-4676-af97-c16fc92d8c20\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"
Jul 29 19:45:12.375861 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.375706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e029d7af-1acb-4676-af97-c16fc92d8c20-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg\" (UID: \"e029d7af-1acb-4676-af97-c16fc92d8c20\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"
Jul 29 19:45:12.376344 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.376326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e029d7af-1acb-4676-af97-c16fc92d8c20-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg\" (UID: \"e029d7af-1acb-4676-af97-c16fc92d8c20\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"
Jul 29 19:45:12.387935 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.387910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cfdx\" (UniqueName: \"kubernetes.io/projected/e029d7af-1acb-4676-af97-c16fc92d8c20-kube-api-access-8cfdx\") pod \"payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg\" (UID: \"e029d7af-1acb-4676-af97-c16fc92d8c20\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"
Jul 29 19:45:12.474149 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.474112    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"
Jul 29 19:45:12.476511 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.476487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shrrx\" (UniqueName: \"kubernetes.io/projected/5edd5028-1cb7-4213-aa68-205703a7e339-kube-api-access-shrrx\") pod \"payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt\" (UID: \"5edd5028-1cb7-4213-aa68-205703a7e339\") " pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"
Jul 29 19:45:12.476606 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.476555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edd5028-1cb7-4213-aa68-205703a7e339-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt\" (UID: \"5edd5028-1cb7-4213-aa68-205703a7e339\") " pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"
Jul 29 19:45:12.577935 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.577887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shrrx\" (UniqueName: \"kubernetes.io/projected/5edd5028-1cb7-4213-aa68-205703a7e339-kube-api-access-shrrx\") pod \"payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt\" (UID: \"5edd5028-1cb7-4213-aa68-205703a7e339\") " pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"
Jul 29 19:45:12.578117 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.577985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edd5028-1cb7-4213-aa68-205703a7e339-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt\" (UID: \"5edd5028-1cb7-4213-aa68-205703a7e339\") " pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"
Jul 29 19:45:12.578964 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.578932    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edd5028-1cb7-4213-aa68-205703a7e339-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt\" (UID: \"5edd5028-1cb7-4213-aa68-205703a7e339\") " pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"
Jul 29 19:45:12.588120 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.588087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shrrx\" (UniqueName: \"kubernetes.io/projected/5edd5028-1cb7-4213-aa68-205703a7e339-kube-api-access-shrrx\") pod \"payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt\" (UID: \"5edd5028-1cb7-4213-aa68-205703a7e339\") " pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"
Jul 29 19:45:12.615541 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.615513    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"]
Jul 29 19:45:12.618181 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:45:12.618148    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode029d7af_1acb_4676_af97_c16fc92d8c20.slice/crio-5778cdd7b68c4ab43ce1e29bdf8931e9b9d8e03ffb9580d85fe1de42c1d50024 WatchSource:0}: Error finding container 5778cdd7b68c4ab43ce1e29bdf8931e9b9d8e03ffb9580d85fe1de42c1d50024: Status 404 returned error can't find the container with id 5778cdd7b68c4ab43ce1e29bdf8931e9b9d8e03ffb9580d85fe1de42c1d50024
Jul 29 19:45:12.625029 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.624994    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"
Jul 29 19:45:12.780418 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.780389    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"]
Jul 29 19:45:12.781502 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:45:12.781478    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edd5028_1cb7_4213_aa68_205703a7e339.slice/crio-f1ec04d8423290cc000a506a8b2fe9391324ebdbeb2940e452220ee32e05998d WatchSource:0}: Error finding container f1ec04d8423290cc000a506a8b2fe9391324ebdbeb2940e452220ee32e05998d: Status 404 returned error can't find the container with id f1ec04d8423290cc000a506a8b2fe9391324ebdbeb2940e452220ee32e05998d
Jul 29 19:45:12.882136 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.882098    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt" event={"ID":"5edd5028-1cb7-4213-aa68-205703a7e339","Type":"ContainerStarted","Data":"8cef3701e28baa5e23e9002672c5238306c179808dac679eaf07b46adf06e351"}
Jul 29 19:45:12.882348 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.882167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt" event={"ID":"5edd5028-1cb7-4213-aa68-205703a7e339","Type":"ContainerStarted","Data":"f1ec04d8423290cc000a506a8b2fe9391324ebdbeb2940e452220ee32e05998d"}
Jul 29 19:45:12.882463 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.882429    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"
Jul 29 19:45:12.884040 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.884006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg" event={"ID":"e029d7af-1acb-4676-af97-c16fc92d8c20","Type":"ContainerStarted","Data":"e9e63ebb1e66e93dba01506337deadd355ad2290f17bad635e301153ff7be9c3"}
Jul 29 19:45:12.884171 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.884066    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"
Jul 29 19:45:12.884171 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.884090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg" event={"ID":"e029d7af-1acb-4676-af97-c16fc92d8c20","Type":"ContainerStarted","Data":"5778cdd7b68c4ab43ce1e29bdf8931e9b9d8e03ffb9580d85fe1de42c1d50024"}
Jul 29 19:45:12.885732 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.885708    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw" event={"ID":"1e264efc-fae3-459d-9a39-cf4c9f8fdae8","Type":"ContainerStarted","Data":"e76c9bff4fb48276eace673d672285c37b09e78b65b59223f9a4276f02dcc4f3"}
Jul 29 19:45:12.885854 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.885835    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:45:12.912395 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.912342    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt" podStartSLOduration=0.912327038 podStartE2EDuration="912.327038ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:12.910314023 +0000 UTC m=+2679.209986525" watchObservedRunningTime="2026-07-29 19:45:12.912327038 +0000 UTC m=+2679.211999537"
Jul 29 19:45:12.973142 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:12.973043    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw" podStartSLOduration=1.371096853 podStartE2EDuration="1.973027195s" totalImagesPullingTime="601.930342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:12.972229464 +0000 UTC m=+2679.271901964" watchObservedRunningTime="2026-07-29 19:45:12.973027195 +0000 UTC m=+2679.272699695"
Jul 29 19:45:13.003439 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:13.003391    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg" podStartSLOduration=1.003374499 podStartE2EDuration="1.003374499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:13.002560681 +0000 UTC m=+2679.302233181" watchObservedRunningTime="2026-07-29 19:45:13.003374499 +0000 UTC m=+2679.303046998"
Jul 29 19:45:18.898236 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:18.898206    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:45:23.894860 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:23.894825    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"
Jul 29 19:45:23.895352 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:23.895046    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"
Jul 29 19:45:34.335003 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:34.334976    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:45:34.342559 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:34.342532    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:45:39.379294 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:39.379264    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"]
Jul 29 19:45:39.384898 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:39.384875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:45:39.388356 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:39.388333    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-808386-serving-cert\""
Jul 29 19:45:39.395610 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:39.395582    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"]
Jul 29 19:45:39.418700 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:39.418661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-maas-api-tls\") pod \"maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn\" (UID: \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:45:39.418700 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:39.418707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxqfv\" (UniqueName: \"kubernetes.io/projected/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-kube-api-access-kxqfv\") pod \"maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn\" (UID: \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:45:39.418949 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:39.418740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn\" (UID: \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:45:39.520221 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:39.520139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-maas-api-tls\") pod \"maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn\" (UID: \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:45:39.520221 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:39.520230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxqfv\" (UniqueName: \"kubernetes.io/projected/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-kube-api-access-kxqfv\") pod \"maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn\" (UID: \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:45:39.520497 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:39.520261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn\" (UID: \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:45:39.520497 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:45:39.520385    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-serving-cert: secret "maas-api-e2e-shared-b-808386-serving-cert" not found
Jul 29 19:45:39.520497 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:45:39.520468    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-maas-api-tls podName:6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8 nodeName:}" failed. No retries permitted until 2026-07-29 19:45:40.020446892 +0000 UTC m=+2706.320119376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-maas-api-tls") pod "maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn" (UID: "6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8") : secret "maas-api-e2e-shared-b-808386-serving-cert" not found
Jul 29 19:45:39.520982 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:39.520959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn\" (UID: \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:45:39.534919 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:39.534891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxqfv\" (UniqueName: \"kubernetes.io/projected/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-kube-api-access-kxqfv\") pod \"maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn\" (UID: \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:45:40.025607 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.025575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-maas-api-tls\") pod \"maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn\" (UID: \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:45:40.028041 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.028022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-maas-api-tls\") pod \"maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn\" (UID: \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:45:40.300099 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.300015    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:45:40.422359 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.422328    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"]
Jul 29 19:45:40.426297 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.426274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"
Jul 29 19:45:40.429352 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.429332    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-808386\""
Jul 29 19:45:40.440297 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.440265    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"]
Jul 29 19:45:40.443004 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.442975    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"]
Jul 29 19:45:40.446476 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.446459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"
Jul 29 19:45:40.449653 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.449634    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-808386-dockercfg-l5l5k\""
Jul 29 19:45:40.462080 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.462051    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"]
Jul 29 19:45:40.464389 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.464355    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"]
Jul 29 19:45:40.464493 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:45:40.464429    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce6316c_2c73_4d6b_8612_ab3d2d7cdfd8.slice/crio-464eb33bde4ae7eda8e0db56a623b37fb525d3557f08fe309945755ad6524a32 WatchSource:0}: Error finding container 464eb33bde4ae7eda8e0db56a623b37fb525d3557f08fe309945755ad6524a32: Status 404 returned error can't find the container with id 464eb33bde4ae7eda8e0db56a623b37fb525d3557f08fe309945755ad6524a32
Jul 29 19:45:40.529836 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.529804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de79cbb9-8dfb-4a36-a5dc-c592668c55bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb\" (UID: \"de79cbb9-8dfb-4a36-a5dc-c592668c55bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"
Jul 29 19:45:40.529997 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.529854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr\" (UID: \"3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"
Jul 29 19:45:40.529997 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.529922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq78j\" (UniqueName: \"kubernetes.io/projected/3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341-kube-api-access-rq78j\") pod \"payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr\" (UID: \"3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"
Jul 29 19:45:40.529997 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.529962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-685pl\" (UniqueName: \"kubernetes.io/projected/de79cbb9-8dfb-4a36-a5dc-c592668c55bb-kube-api-access-685pl\") pod \"payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb\" (UID: \"de79cbb9-8dfb-4a36-a5dc-c592668c55bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"
Jul 29 19:45:40.631308 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.631278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr\" (UID: \"3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"
Jul 29 19:45:40.631479 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.631335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq78j\" (UniqueName: \"kubernetes.io/projected/3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341-kube-api-access-rq78j\") pod \"payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr\" (UID: \"3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"
Jul 29 19:45:40.631479 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.631357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-685pl\" (UniqueName: \"kubernetes.io/projected/de79cbb9-8dfb-4a36-a5dc-c592668c55bb-kube-api-access-685pl\") pod \"payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb\" (UID: \"de79cbb9-8dfb-4a36-a5dc-c592668c55bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"
Jul 29 19:45:40.631479 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.631409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de79cbb9-8dfb-4a36-a5dc-c592668c55bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb\" (UID: \"de79cbb9-8dfb-4a36-a5dc-c592668c55bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"
Jul 29 19:45:40.632052 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.632030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr\" (UID: \"3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"
Jul 29 19:45:40.632107 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.632090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de79cbb9-8dfb-4a36-a5dc-c592668c55bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb\" (UID: \"de79cbb9-8dfb-4a36-a5dc-c592668c55bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"
Jul 29 19:45:40.640985 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.640948    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq78j\" (UniqueName: \"kubernetes.io/projected/3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341-kube-api-access-rq78j\") pod \"payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr\" (UID: \"3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"
Jul 29 19:45:40.641124 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.641039    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-685pl\" (UniqueName: \"kubernetes.io/projected/de79cbb9-8dfb-4a36-a5dc-c592668c55bb-kube-api-access-685pl\") pod \"payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb\" (UID: \"de79cbb9-8dfb-4a36-a5dc-c592668c55bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"
Jul 29 19:45:40.738429 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.738398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"
Jul 29 19:45:40.756868 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.756836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"
Jul 29 19:45:40.909879 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.909850    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"]
Jul 29 19:45:40.911644 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:45:40.911608    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae5ae0d_2a0c_4e9d_957a_0d24dd5df341.slice/crio-fb7cc5b202f25606313897aa330720e79145211b72a513257a36f23f63f13de2 WatchSource:0}: Error finding container fb7cc5b202f25606313897aa330720e79145211b72a513257a36f23f63f13de2: Status 404 returned error can't find the container with id fb7cc5b202f25606313897aa330720e79145211b72a513257a36f23f63f13de2
Jul 29 19:45:40.942114 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:40.942081    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"]
Jul 29 19:45:40.943756 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:45:40.943725    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde79cbb9_8dfb_4a36_a5dc_c592668c55bb.slice/crio-4b44382a971c3e62a3ff588d7439141966bc7840dcd5fb1ab3a3e75de174710d WatchSource:0}: Error finding container 4b44382a971c3e62a3ff588d7439141966bc7840dcd5fb1ab3a3e75de174710d: Status 404 returned error can't find the container with id 4b44382a971c3e62a3ff588d7439141966bc7840dcd5fb1ab3a3e75de174710d
Jul 29 19:45:41.031946 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:41.031906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr" event={"ID":"3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341","Type":"ContainerStarted","Data":"dd854b18d00e579aca1250be87abd091db9b298ad8b7fdaf0b2b5fe3920e518a"}
Jul 29 19:45:41.032132 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:41.031969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr" event={"ID":"3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341","Type":"ContainerStarted","Data":"fb7cc5b202f25606313897aa330720e79145211b72a513257a36f23f63f13de2"}
Jul 29 19:45:41.032132 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:41.032081    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"
Jul 29 19:45:41.033497 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:41.033471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb" event={"ID":"de79cbb9-8dfb-4a36-a5dc-c592668c55bb","Type":"ContainerStarted","Data":"46f96f319974d530f38fc2869f438da6c55bed698beff08b3ad43920526bfba3"}
Jul 29 19:45:41.033635 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:41.033530    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"
Jul 29 19:45:41.033635 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:41.033547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb" event={"ID":"de79cbb9-8dfb-4a36-a5dc-c592668c55bb","Type":"ContainerStarted","Data":"4b44382a971c3e62a3ff588d7439141966bc7840dcd5fb1ab3a3e75de174710d"}
Jul 29 19:45:41.035096 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:41.035075    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn" event={"ID":"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8","Type":"ContainerStarted","Data":"bc2d86fd33546e514b8e12af625fc653bfa85a0c0b8a7d6653add1beaef3a9bb"}
Jul 29 19:45:41.035235 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:41.035118    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn" event={"ID":"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8","Type":"ContainerStarted","Data":"464eb33bde4ae7eda8e0db56a623b37fb525d3557f08fe309945755ad6524a32"}
Jul 29 19:45:41.035235 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:41.035150    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:45:41.069463 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:41.069405    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr" podStartSLOduration=1.069388566 podStartE2EDuration="1.069388566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:41.067327048 +0000 UTC m=+2707.366999544" watchObservedRunningTime="2026-07-29 19:45:41.069388566 +0000 UTC m=+2707.369061044"
Jul 29 19:45:41.099477 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:41.099429    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb" podStartSLOduration=1.099413061 podStartE2EDuration="1.099413061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:41.099292284 +0000 UTC m=+2707.398964784" watchObservedRunningTime="2026-07-29 19:45:41.099413061 +0000 UTC m=+2707.399085560"
Jul 29 19:45:41.123054 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:41.122996    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn" podStartSLOduration=1.732534698 podStartE2EDuration="2.12297579s" totalImagesPullingTime="390.441092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:41.121089503 +0000 UTC m=+2707.420762029" watchObservedRunningTime="2026-07-29 19:45:41.12297579 +0000 UTC m=+2707.422648291"
Jul 29 19:45:47.045880 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:47.045851    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:45:52.043591 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:52.043560    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"
Jul 29 19:45:52.044045 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:45:52.043631    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"
Jul 29 19:46:43.203397 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.203364    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"]
Jul 29 19:46:43.207210 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.207175    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.210582 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.210557    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-808386\"/\"openshift-service-ca.crt\""
Jul 29 19:46:43.210707 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.210614    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-808386\"/\"kube-root-ca.crt\""
Jul 29 19:46:43.211868 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.211848    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-808386\"/\"default-dockercfg-vtvdg\""
Jul 29 19:46:43.211979 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.211915    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-808386\"/\"auth-test-model-808386-kserve-self-signed-certs\""
Jul 29 19:46:43.217964 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.217941    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"]
Jul 29 19:46:43.302028 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.301988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-dshm\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.302028 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.302032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-home\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.302326 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.302057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-model-cache\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.302326 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.302186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-tmp-dir\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.302326 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.302246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2e7935d-1629-4760-a32c-39c89a9f1b50-tls-certs\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.302469 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.302329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85x7q\" (UniqueName: \"kubernetes.io/projected/a2e7935d-1629-4760-a32c-39c89a9f1b50-kube-api-access-85x7q\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.403863 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.403832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-tmp-dir\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.403863 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.403867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2e7935d-1629-4760-a32c-39c89a9f1b50-tls-certs\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.404106 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.403906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85x7q\" (UniqueName: \"kubernetes.io/projected/a2e7935d-1629-4760-a32c-39c89a9f1b50-kube-api-access-85x7q\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.404106 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.403966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-dshm\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.404106 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.403993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-home\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.404106 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.404012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-model-cache\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.404357 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.404314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-tmp-dir\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.404425 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.404402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-model-cache\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.404473 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.404424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-home\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.406275 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.406254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-dshm\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.406514 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.406496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2e7935d-1629-4760-a32c-39c89a9f1b50-tls-certs\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.413513 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.413486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85x7q\" (UniqueName: \"kubernetes.io/projected/a2e7935d-1629-4760-a32c-39c89a9f1b50-kube-api-access-85x7q\") pod \"auth-test-model-808386-kserve-6dd67b4b8c-66kj6\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") " pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.518533 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.518440    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:43.654339 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:43.654316    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"]
Jul 29 19:46:43.656412 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:46:43.656383    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2e7935d_1629_4760_a32c_39c89a9f1b50.slice/crio-68a54e6ab1e8fdbfabb48cee59f595745d0bbb3843701f8c7b92a38a2a750cda WatchSource:0}: Error finding container 68a54e6ab1e8fdbfabb48cee59f595745d0bbb3843701f8c7b92a38a2a750cda: Status 404 returned error can't find the container with id 68a54e6ab1e8fdbfabb48cee59f595745d0bbb3843701f8c7b92a38a2a750cda
Jul 29 19:46:44.352797 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:44.352763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6" event={"ID":"a2e7935d-1629-4760-a32c-39c89a9f1b50","Type":"ContainerStarted","Data":"64aaebbe80911d3ec29be02b3f6a1fd0e4a71399a34779e4eb4a52ebf37092c9"}
Jul 29 19:46:44.353312 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:44.352815    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6" event={"ID":"a2e7935d-1629-4760-a32c-39c89a9f1b50","Type":"ContainerStarted","Data":"68a54e6ab1e8fdbfabb48cee59f595745d0bbb3843701f8c7b92a38a2a750cda"}
Jul 29 19:46:44.370848 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:44.370798    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6" podStartSLOduration=1.370782677 podStartE2EDuration="1.370782677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:46:44.370129379 +0000 UTC m=+2770.669801878" watchObservedRunningTime="2026-07-29 19:46:44.370782677 +0000 UTC m=+2770.670455175"
Jul 29 19:46:53.518818 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:53.518773    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:53.519351 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:53.518841    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:53.530889 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:53.530866    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:46:54.412931 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:46:54.412853    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:50:34.367712 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:50:34.367677    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:50:34.375923 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:50:34.375894    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:50:48.215505 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:50:48.215471    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"]
Jul 29 19:51:03.235584 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.235539    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6" podUID="a2e7935d-1629-4760-a32c-39c89a9f1b50" containerName="main" containerID="cri-o://64aaebbe80911d3ec29be02b3f6a1fd0e4a71399a34779e4eb4a52ebf37092c9" gracePeriod=45
Jul 29 19:51:03.493893 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.493823    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:51:03.554174 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.554127    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-model-cache\") pod \"a2e7935d-1629-4760-a32c-39c89a9f1b50\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") "
Jul 29 19:51:03.554374 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.554218    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-tmp-dir\") pod \"a2e7935d-1629-4760-a32c-39c89a9f1b50\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") "
Jul 29 19:51:03.554374 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.554247    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85x7q\" (UniqueName: \"kubernetes.io/projected/a2e7935d-1629-4760-a32c-39c89a9f1b50-kube-api-access-85x7q\") pod \"a2e7935d-1629-4760-a32c-39c89a9f1b50\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") "
Jul 29 19:51:03.554374 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.554265    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-home\") pod \"a2e7935d-1629-4760-a32c-39c89a9f1b50\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") "
Jul 29 19:51:03.554374 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.554308    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2e7935d-1629-4760-a32c-39c89a9f1b50-tls-certs\") pod \"a2e7935d-1629-4760-a32c-39c89a9f1b50\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") "
Jul 29 19:51:03.554374 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.554344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-dshm\") pod \"a2e7935d-1629-4760-a32c-39c89a9f1b50\" (UID: \"a2e7935d-1629-4760-a32c-39c89a9f1b50\") "
Jul 29 19:51:03.554633 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.554463    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2e7935d-1629-4760-a32c-39c89a9f1b50" (UID: "a2e7935d-1629-4760-a32c-39c89a9f1b50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:51:03.554633 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.554546    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-home" (OuterVolumeSpecName: "home") pod "a2e7935d-1629-4760-a32c-39c89a9f1b50" (UID: "a2e7935d-1629-4760-a32c-39c89a9f1b50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:51:03.554633 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.554554    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2e7935d-1629-4760-a32c-39c89a9f1b50" (UID: "a2e7935d-1629-4760-a32c-39c89a9f1b50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:51:03.554797 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.554667    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-model-cache\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:51:03.554797 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.554686    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-tmp-dir\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:51:03.554797 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.554695    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-home\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:51:03.556767 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.556728    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2e7935d-1629-4760-a32c-39c89a9f1b50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2e7935d-1629-4760-a32c-39c89a9f1b50" (UID: "a2e7935d-1629-4760-a32c-39c89a9f1b50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:51:03.556888 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.556731    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-dshm" (OuterVolumeSpecName: "dshm") pod "a2e7935d-1629-4760-a32c-39c89a9f1b50" (UID: "a2e7935d-1629-4760-a32c-39c89a9f1b50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:51:03.556888 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.556778    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e7935d-1629-4760-a32c-39c89a9f1b50-kube-api-access-85x7q" (OuterVolumeSpecName: "kube-api-access-85x7q") pod "a2e7935d-1629-4760-a32c-39c89a9f1b50" (UID: "a2e7935d-1629-4760-a32c-39c89a9f1b50"). InnerVolumeSpecName "kube-api-access-85x7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:51:03.655882 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.655834    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2e7935d-1629-4760-a32c-39c89a9f1b50-tls-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:51:03.655882 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.655871    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2e7935d-1629-4760-a32c-39c89a9f1b50-dshm\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:51:03.655882 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.655885    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85x7q\" (UniqueName: \"kubernetes.io/projected/a2e7935d-1629-4760-a32c-39c89a9f1b50-kube-api-access-85x7q\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:51:03.705995 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.705959    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2e7935d-1629-4760-a32c-39c89a9f1b50" containerID="64aaebbe80911d3ec29be02b3f6a1fd0e4a71399a34779e4eb4a52ebf37092c9" exitCode=0
Jul 29 19:51:03.706240 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.706006    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"
Jul 29 19:51:03.706240 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.706044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6" event={"ID":"a2e7935d-1629-4760-a32c-39c89a9f1b50","Type":"ContainerDied","Data":"64aaebbe80911d3ec29be02b3f6a1fd0e4a71399a34779e4eb4a52ebf37092c9"}
Jul 29 19:51:03.706240 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.706085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6" event={"ID":"a2e7935d-1629-4760-a32c-39c89a9f1b50","Type":"ContainerDied","Data":"68a54e6ab1e8fdbfabb48cee59f595745d0bbb3843701f8c7b92a38a2a750cda"}
Jul 29 19:51:03.706240 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.706102    2579 scope.go:117] "RemoveContainer" containerID="64aaebbe80911d3ec29be02b3f6a1fd0e4a71399a34779e4eb4a52ebf37092c9"
Jul 29 19:51:03.715852 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.715830    2579 scope.go:117] "RemoveContainer" containerID="64aaebbe80911d3ec29be02b3f6a1fd0e4a71399a34779e4eb4a52ebf37092c9"
Jul 29 19:51:03.716134 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:51:03.716112    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64aaebbe80911d3ec29be02b3f6a1fd0e4a71399a34779e4eb4a52ebf37092c9\": container with ID starting with 64aaebbe80911d3ec29be02b3f6a1fd0e4a71399a34779e4eb4a52ebf37092c9 not found: ID does not exist" containerID="64aaebbe80911d3ec29be02b3f6a1fd0e4a71399a34779e4eb4a52ebf37092c9"
Jul 29 19:51:03.716235 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.716143    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64aaebbe80911d3ec29be02b3f6a1fd0e4a71399a34779e4eb4a52ebf37092c9"} err="failed to get container status \"64aaebbe80911d3ec29be02b3f6a1fd0e4a71399a34779e4eb4a52ebf37092c9\": rpc error: code = NotFound desc = could not find container \"64aaebbe80911d3ec29be02b3f6a1fd0e4a71399a34779e4eb4a52ebf37092c9\": container with ID starting with 64aaebbe80911d3ec29be02b3f6a1fd0e4a71399a34779e4eb4a52ebf37092c9 not found: ID does not exist"
Jul 29 19:51:03.728443 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.728403    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"]
Jul 29 19:51:03.734686 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:03.734652    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-808386/auth-test-model-808386-kserve-6dd67b4b8c-66kj6"]
Jul 29 19:51:04.196319 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:51:04.196271    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2e7935d-1629-4760-a32c-39c89a9f1b50" path="/var/lib/kubelet/pods/a2e7935d-1629-4760-a32c-39c89a9f1b50/volumes"
Jul 29 19:52:34.218609 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.218575    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"]
Jul 29 19:52:34.219078 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.218973    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2e7935d-1629-4760-a32c-39c89a9f1b50" containerName="main"
Jul 29 19:52:34.219078 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.218984    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2e7935d-1629-4760-a32c-39c89a9f1b50" containerName="main"
Jul 29 19:52:34.219078 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.219053    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2e7935d-1629-4760-a32c-39c89a9f1b50" containerName="main"
Jul 29 19:52:34.222160 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.222143    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.226903 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.226877    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8e241a\"/\"kube-root-ca.crt\""
Jul 29 19:52:34.227067 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.226922    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8e241a\"/\"default-dockercfg-s79p8\""
Jul 29 19:52:34.227067 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.226928    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8e241a\"/\"rate-test-model-8e241a-kserve-self-signed-certs\""
Jul 29 19:52:34.229441 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.229416    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8e241a\"/\"openshift-service-ca.crt\""
Jul 29 19:52:34.238483 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.238446    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"]
Jul 29 19:52:34.340150 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.340111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-home\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.340150 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.340147    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-tmp-dir\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.340442 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.340170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvfmp\" (UniqueName: \"kubernetes.io/projected/d4749739-4320-40af-9230-7783f792a7e5-kube-api-access-dvfmp\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.340442 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.340247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-dshm\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.340442 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.340265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-model-cache\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.340442 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.340391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4749739-4320-40af-9230-7783f792a7e5-tls-certs\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.441912 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.441878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-model-cache\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.442124 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.441946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4749739-4320-40af-9230-7783f792a7e5-tls-certs\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.442124 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.441973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-home\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.442124 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.441988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-tmp-dir\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.442353 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.442128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvfmp\" (UniqueName: \"kubernetes.io/projected/d4749739-4320-40af-9230-7783f792a7e5-kube-api-access-dvfmp\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.442353 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.442244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-dshm\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.442455 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.442363    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-model-cache\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.442455 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.442427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-tmp-dir\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.442550 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.442452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-home\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.444531 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.444502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-dshm\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.444727 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.444712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4749739-4320-40af-9230-7783f792a7e5-tls-certs\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.453071 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.453044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvfmp\" (UniqueName: \"kubernetes.io/projected/d4749739-4320-40af-9230-7783f792a7e5-kube-api-access-dvfmp\") pod \"rate-test-model-8e241a-kserve-7875f65f9f-4ws68\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") " pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.531282 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.531149    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:34.684201 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:34.684160    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"]
Jul 29 19:52:34.685713 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:52:34.685687    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4749739_4320_40af_9230_7783f792a7e5.slice/crio-27aef42c6b4af760cc49883f4a6df8df37feadbf230cc1324e3f29265f73c296 WatchSource:0}: Error finding container 27aef42c6b4af760cc49883f4a6df8df37feadbf230cc1324e3f29265f73c296: Status 404 returned error can't find the container with id 27aef42c6b4af760cc49883f4a6df8df37feadbf230cc1324e3f29265f73c296
Jul 29 19:52:35.189642 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:35.189595    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68" event={"ID":"d4749739-4320-40af-9230-7783f792a7e5","Type":"ContainerStarted","Data":"78201861cca328105ec1f0a3bfc585dab315c48c4cebddfc724056e153daa815"}
Jul 29 19:52:35.189642 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:35.189649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68" event={"ID":"d4749739-4320-40af-9230-7783f792a7e5","Type":"ContainerStarted","Data":"27aef42c6b4af760cc49883f4a6df8df37feadbf230cc1324e3f29265f73c296"}
Jul 29 19:52:35.210797 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:35.210746    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68" podStartSLOduration=1.210723771 podStartE2EDuration="1.210723771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:52:35.207758194 +0000 UTC m=+3121.507430722" watchObservedRunningTime="2026-07-29 19:52:35.210723771 +0000 UTC m=+3121.510396270"
Jul 29 19:52:44.531583 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:44.531549    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:44.532005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:44.531600    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:44.544982 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:44.544949    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:52:45.252606 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:52:45.252569    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:54:07.626032 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:07.625996    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"]
Jul 29 19:54:12.672408 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:12.672368    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7638-140520098585776"]
Jul 29 19:54:12.675970 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:12.675953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098585776"
Jul 29 19:54:12.679023 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:12.678999    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lhphn\""
Jul 29 19:54:12.679991 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:12.679967    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7638-140520098585776"]
Jul 29 19:54:12.773755 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:12.773717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2thd\" (UniqueName: \"kubernetes.io/projected/d0c7d438-8086-4ae2-b0cc-ff19951e0a41-kube-api-access-w2thd\") pod \"test-curl-7638-140520098585776\" (UID: \"d0c7d438-8086-4ae2-b0cc-ff19951e0a41\") " pod="opendatahub/test-curl-7638-140520098585776"
Jul 29 19:54:12.875092 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:12.875053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2thd\" (UniqueName: \"kubernetes.io/projected/d0c7d438-8086-4ae2-b0cc-ff19951e0a41-kube-api-access-w2thd\") pod \"test-curl-7638-140520098585776\" (UID: \"d0c7d438-8086-4ae2-b0cc-ff19951e0a41\") " pod="opendatahub/test-curl-7638-140520098585776"
Jul 29 19:54:12.884273 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:12.884244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2thd\" (UniqueName: \"kubernetes.io/projected/d0c7d438-8086-4ae2-b0cc-ff19951e0a41-kube-api-access-w2thd\") pod \"test-curl-7638-140520098585776\" (UID: \"d0c7d438-8086-4ae2-b0cc-ff19951e0a41\") " pod="opendatahub/test-curl-7638-140520098585776"
Jul 29 19:54:12.987795 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:12.987698    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098585776"
Jul 29 19:54:13.131657 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:13.131631    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7638-140520098585776"]
Jul 29 19:54:13.133944 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:54:13.133916    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0c7d438_8086_4ae2_b0cc_ff19951e0a41.slice/crio-9e91797496edc1519209b610f4975aeb2a66fc6456ed296c5e8bd2a7ca605762 WatchSource:0}: Error finding container 9e91797496edc1519209b610f4975aeb2a66fc6456ed296c5e8bd2a7ca605762: Status 404 returned error can't find the container with id 9e91797496edc1519209b610f4975aeb2a66fc6456ed296c5e8bd2a7ca605762
Jul 29 19:54:13.135799 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:13.135780    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:54:13.709268 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:13.709227    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7638-140520098585776" event={"ID":"d0c7d438-8086-4ae2-b0cc-ff19951e0a41","Type":"ContainerStarted","Data":"9e91797496edc1519209b610f4975aeb2a66fc6456ed296c5e8bd2a7ca605762"}
Jul 29 19:54:14.715757 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:14.715718    2579 generic.go:356] "Generic (PLEG): container finished" podID="d0c7d438-8086-4ae2-b0cc-ff19951e0a41" containerID="0b4feba256bbefea96c0ae327cd941e4b69bfa0687eaec2a9da8dbc89ab1583c" exitCode=0
Jul 29 19:54:14.716162 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:14.715811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7638-140520098585776" event={"ID":"d0c7d438-8086-4ae2-b0cc-ff19951e0a41","Type":"ContainerDied","Data":"0b4feba256bbefea96c0ae327cd941e4b69bfa0687eaec2a9da8dbc89ab1583c"}
Jul 29 19:54:15.865667 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:15.865640    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098585776"
Jul 29 19:54:15.917669 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:15.917641    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7638-140520098585776_d0c7d438-8086-4ae2-b0cc-ff19951e0a41/test-curl-7638-140520098585776/0.log"
Jul 29 19:54:15.941468 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:15.941438    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7638-140520098585776"]
Jul 29 19:54:15.945917 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:15.945888    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7638-140520098585776"]
Jul 29 19:54:16.004359 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:16.004257    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2thd\" (UniqueName: \"kubernetes.io/projected/d0c7d438-8086-4ae2-b0cc-ff19951e0a41-kube-api-access-w2thd\") pod \"d0c7d438-8086-4ae2-b0cc-ff19951e0a41\" (UID: \"d0c7d438-8086-4ae2-b0cc-ff19951e0a41\") "
Jul 29 19:54:16.006584 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:16.006559    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c7d438-8086-4ae2-b0cc-ff19951e0a41-kube-api-access-w2thd" (OuterVolumeSpecName: "kube-api-access-w2thd") pod "d0c7d438-8086-4ae2-b0cc-ff19951e0a41" (UID: "d0c7d438-8086-4ae2-b0cc-ff19951e0a41"). InnerVolumeSpecName "kube-api-access-w2thd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:54:16.105750 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:16.105713    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2thd\" (UniqueName: \"kubernetes.io/projected/d0c7d438-8086-4ae2-b0cc-ff19951e0a41-kube-api-access-w2thd\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:54:16.196586 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:16.196549    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0c7d438-8086-4ae2-b0cc-ff19951e0a41" path="/var/lib/kubelet/pods/d0c7d438-8086-4ae2-b0cc-ff19951e0a41/volumes"
Jul 29 19:54:16.728930 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:16.728903    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098585776"
Jul 29 19:54:16.728930 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:16.728919    2579 scope.go:117] "RemoveContainer" containerID="0b4feba256bbefea96c0ae327cd941e4b69bfa0687eaec2a9da8dbc89ab1583c"
Jul 29 19:54:22.646401 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:22.646351    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68" podUID="d4749739-4320-40af-9230-7783f792a7e5" containerName="main" containerID="cri-o://78201861cca328105ec1f0a3bfc585dab315c48c4cebddfc724056e153daa815" gracePeriod=45
Jul 29 19:54:22.763781 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:22.763748    2579 generic.go:356] "Generic (PLEG): container finished" podID="d4749739-4320-40af-9230-7783f792a7e5" containerID="78201861cca328105ec1f0a3bfc585dab315c48c4cebddfc724056e153daa815" exitCode=0
Jul 29 19:54:22.763924 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:22.763821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68" event={"ID":"d4749739-4320-40af-9230-7783f792a7e5","Type":"ContainerDied","Data":"78201861cca328105ec1f0a3bfc585dab315c48c4cebddfc724056e153daa815"}
Jul 29 19:54:22.904762 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:22.904694    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:54:23.079264 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.079226    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-tmp-dir\") pod \"d4749739-4320-40af-9230-7783f792a7e5\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") "
Jul 29 19:54:23.079264 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.079263    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4749739-4320-40af-9230-7783f792a7e5-tls-certs\") pod \"d4749739-4320-40af-9230-7783f792a7e5\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") "
Jul 29 19:54:23.079531 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.079284    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-home\") pod \"d4749739-4320-40af-9230-7783f792a7e5\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") "
Jul 29 19:54:23.079531 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.079329    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvfmp\" (UniqueName: \"kubernetes.io/projected/d4749739-4320-40af-9230-7783f792a7e5-kube-api-access-dvfmp\") pod \"d4749739-4320-40af-9230-7783f792a7e5\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") "
Jul 29 19:54:23.079531 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.079449    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-dshm\") pod \"d4749739-4320-40af-9230-7783f792a7e5\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") "
Jul 29 19:54:23.079689 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.079541    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4749739-4320-40af-9230-7783f792a7e5" (UID: "d4749739-4320-40af-9230-7783f792a7e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:54:23.079689 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.079569    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-model-cache\") pod \"d4749739-4320-40af-9230-7783f792a7e5\" (UID: \"d4749739-4320-40af-9230-7783f792a7e5\") "
Jul 29 19:54:23.079689 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.079590    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-home" (OuterVolumeSpecName: "home") pod "d4749739-4320-40af-9230-7783f792a7e5" (UID: "d4749739-4320-40af-9230-7783f792a7e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:54:23.079806 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.079793    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4749739-4320-40af-9230-7783f792a7e5" (UID: "d4749739-4320-40af-9230-7783f792a7e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:54:23.079952 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.079924    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-tmp-dir\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:54:23.079952 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.079950    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-home\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:54:23.080350 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.079964    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-model-cache\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:54:23.081809 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.081774    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-dshm" (OuterVolumeSpecName: "dshm") pod "d4749739-4320-40af-9230-7783f792a7e5" (UID: "d4749739-4320-40af-9230-7783f792a7e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:54:23.081907 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.081833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4749739-4320-40af-9230-7783f792a7e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4749739-4320-40af-9230-7783f792a7e5" (UID: "d4749739-4320-40af-9230-7783f792a7e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:54:23.081907 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.081843    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4749739-4320-40af-9230-7783f792a7e5-kube-api-access-dvfmp" (OuterVolumeSpecName: "kube-api-access-dvfmp") pod "d4749739-4320-40af-9230-7783f792a7e5" (UID: "d4749739-4320-40af-9230-7783f792a7e5"). InnerVolumeSpecName "kube-api-access-dvfmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:54:23.180765 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.180682    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4749739-4320-40af-9230-7783f792a7e5-tls-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:54:23.180765 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.180709    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvfmp\" (UniqueName: \"kubernetes.io/projected/d4749739-4320-40af-9230-7783f792a7e5-kube-api-access-dvfmp\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:54:23.180765 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.180718    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4749739-4320-40af-9230-7783f792a7e5-dshm\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:54:23.773837 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.773786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68" event={"ID":"d4749739-4320-40af-9230-7783f792a7e5","Type":"ContainerDied","Data":"27aef42c6b4af760cc49883f4a6df8df37feadbf230cc1324e3f29265f73c296"}
Jul 29 19:54:23.774359 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.773856    2579 scope.go:117] "RemoveContainer" containerID="78201861cca328105ec1f0a3bfc585dab315c48c4cebddfc724056e153daa815"
Jul 29 19:54:23.774359 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.774053    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"
Jul 29 19:54:23.797835 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.797801    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"]
Jul 29 19:54:23.801532 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:23.801500    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8e241a/rate-test-model-8e241a-kserve-7875f65f9f-4ws68"]
Jul 29 19:54:24.196935 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:24.196897    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4749739-4320-40af-9230-7783f792a7e5" path="/var/lib/kubelet/pods/d4749739-4320-40af-9230-7783f792a7e5/volumes"
Jul 29 19:54:24.742568 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:24.742533    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7638-140520098651136"]
Jul 29 19:54:24.743027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:24.743009    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4749739-4320-40af-9230-7783f792a7e5" containerName="main"
Jul 29 19:54:24.743126 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:24.743029    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4749739-4320-40af-9230-7783f792a7e5" containerName="main"
Jul 29 19:54:24.743126 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:24.743065    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c7d438-8086-4ae2-b0cc-ff19951e0a41" containerName="test-curl-7638-140520098585776"
Jul 29 19:54:24.743126 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:24.743074    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c7d438-8086-4ae2-b0cc-ff19951e0a41" containerName="test-curl-7638-140520098585776"
Jul 29 19:54:24.743340 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:24.743166    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c7d438-8086-4ae2-b0cc-ff19951e0a41" containerName="test-curl-7638-140520098585776"
Jul 29 19:54:24.743340 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:24.743181    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4749739-4320-40af-9230-7783f792a7e5" containerName="main"
Jul 29 19:54:24.747747 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:24.747722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098651136"
Jul 29 19:54:24.750926 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:24.750900    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lhphn\""
Jul 29 19:54:24.752452 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:24.752421    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7638-140520098651136"]
Jul 29 19:54:24.897857 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:24.897814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8fjs\" (UniqueName: \"kubernetes.io/projected/66aa375a-84a5-467b-8675-c94cc669d1fd-kube-api-access-h8fjs\") pod \"test-curl-7638-140520098651136\" (UID: \"66aa375a-84a5-467b-8675-c94cc669d1fd\") " pod="opendatahub/test-curl-7638-140520098651136"
Jul 29 19:54:24.998738 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:24.998644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fjs\" (UniqueName: \"kubernetes.io/projected/66aa375a-84a5-467b-8675-c94cc669d1fd-kube-api-access-h8fjs\") pod \"test-curl-7638-140520098651136\" (UID: \"66aa375a-84a5-467b-8675-c94cc669d1fd\") " pod="opendatahub/test-curl-7638-140520098651136"
Jul 29 19:54:25.008843 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:25.008809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8fjs\" (UniqueName: \"kubernetes.io/projected/66aa375a-84a5-467b-8675-c94cc669d1fd-kube-api-access-h8fjs\") pod \"test-curl-7638-140520098651136\" (UID: \"66aa375a-84a5-467b-8675-c94cc669d1fd\") " pod="opendatahub/test-curl-7638-140520098651136"
Jul 29 19:54:25.058918 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:25.058877    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098651136"
Jul 29 19:54:25.205897 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:25.205862    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7638-140520098651136"]
Jul 29 19:54:25.208848 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:54:25.208820    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66aa375a_84a5_467b_8675_c94cc669d1fd.slice/crio-5d1662da2674611b75b4fdeeaca47d1d8833b4b2d4af546e64e546d72f05c5f6 WatchSource:0}: Error finding container 5d1662da2674611b75b4fdeeaca47d1d8833b4b2d4af546e64e546d72f05c5f6: Status 404 returned error can't find the container with id 5d1662da2674611b75b4fdeeaca47d1d8833b4b2d4af546e64e546d72f05c5f6
Jul 29 19:54:25.788362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:25.788269    2579 generic.go:356] "Generic (PLEG): container finished" podID="66aa375a-84a5-467b-8675-c94cc669d1fd" containerID="a3aa249be436127ad35b6b3a28c7e1cdbc4bf90fbf653f537294ae57fa483b6d" exitCode=0
Jul 29 19:54:25.788362 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:25.788340    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7638-140520098651136" event={"ID":"66aa375a-84a5-467b-8675-c94cc669d1fd","Type":"ContainerDied","Data":"a3aa249be436127ad35b6b3a28c7e1cdbc4bf90fbf653f537294ae57fa483b6d"}
Jul 29 19:54:25.788575 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:25.788371    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7638-140520098651136" event={"ID":"66aa375a-84a5-467b-8675-c94cc669d1fd","Type":"ContainerStarted","Data":"5d1662da2674611b75b4fdeeaca47d1d8833b4b2d4af546e64e546d72f05c5f6"}
Jul 29 19:54:26.933231 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:26.933205    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098651136"
Jul 29 19:54:26.983176 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:26.983145    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7638-140520098651136_66aa375a-84a5-467b-8675-c94cc669d1fd/test-curl-7638-140520098651136/0.log"
Jul 29 19:54:27.018450 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:27.018412    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7638-140520098651136"]
Jul 29 19:54:27.022317 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:27.022288    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7638-140520098651136"]
Jul 29 19:54:27.117694 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:27.117607    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8fjs\" (UniqueName: \"kubernetes.io/projected/66aa375a-84a5-467b-8675-c94cc669d1fd-kube-api-access-h8fjs\") pod \"66aa375a-84a5-467b-8675-c94cc669d1fd\" (UID: \"66aa375a-84a5-467b-8675-c94cc669d1fd\") "
Jul 29 19:54:27.119920 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:27.119892    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66aa375a-84a5-467b-8675-c94cc669d1fd-kube-api-access-h8fjs" (OuterVolumeSpecName: "kube-api-access-h8fjs") pod "66aa375a-84a5-467b-8675-c94cc669d1fd" (UID: "66aa375a-84a5-467b-8675-c94cc669d1fd"). InnerVolumeSpecName "kube-api-access-h8fjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:54:27.218328 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:27.218294    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8fjs\" (UniqueName: \"kubernetes.io/projected/66aa375a-84a5-467b-8675-c94cc669d1fd-kube-api-access-h8fjs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:54:27.800574 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:27.800543    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098651136"
Jul 29 19:54:27.800787 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:27.800578    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d1662da2674611b75b4fdeeaca47d1d8833b4b2d4af546e64e546d72f05c5f6"
Jul 29 19:54:27.813232 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:54:27.812918    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7638-140520098651136\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="66aa375a-84a5-467b-8675-c94cc669d1fd" pod="opendatahub/test-curl-7638-140520098651136"
Jul 29 19:54:28.196667 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:28.196631    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66aa375a-84a5-467b-8675-c94cc669d1fd" path="/var/lib/kubelet/pods/66aa375a-84a5-467b-8675-c94cc669d1fd/volumes"
Jul 29 19:54:33.750681 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.750645    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"]
Jul 29 19:54:33.751138 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.751116    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66aa375a-84a5-467b-8675-c94cc669d1fd" containerName="test-curl-7638-140520098651136"
Jul 29 19:54:33.751138 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.751132    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="66aa375a-84a5-467b-8675-c94cc669d1fd" containerName="test-curl-7638-140520098651136"
Jul 29 19:54:33.751320 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.751217    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="66aa375a-84a5-467b-8675-c94cc669d1fd" containerName="test-curl-7638-140520098651136"
Jul 29 19:54:33.755121 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.755102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:54:33.758434 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.758412    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9d9f53-serving-cert\""
Jul 29 19:54:33.767386 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.767358    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"]
Jul 29 19:54:33.780834 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.780799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx5xn\" (UniqueName: \"kubernetes.io/projected/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-kube-api-access-kx5xn\") pod \"maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb\" (UID: \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:54:33.780998 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.780878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb\" (UID: \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:54:33.781087 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.780989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb\" (UID: \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:54:33.882429 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.882386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb\" (UID: \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:54:33.882654 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.882479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx5xn\" (UniqueName: \"kubernetes.io/projected/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-kube-api-access-kx5xn\") pod \"maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb\" (UID: \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:54:33.882654 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.882531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb\" (UID: \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:54:33.882654 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:54:33.882548    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-serving-cert: secret "maas-api-e2e-ipp-a-9d9f53-serving-cert" not found
Jul 29 19:54:33.882654 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:54:33.882620    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-maas-api-tls podName:519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31 nodeName:}" failed. No retries permitted until 2026-07-29 19:54:34.382603959 +0000 UTC m=+3240.682276441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-maas-api-tls") pod "maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb" (UID: "519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31") : secret "maas-api-e2e-ipp-a-9d9f53-serving-cert" not found
Jul 29 19:54:33.883234 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.883212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb\" (UID: \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:54:33.895208 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:33.895164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx5xn\" (UniqueName: \"kubernetes.io/projected/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-kube-api-access-kx5xn\") pod \"maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb\" (UID: \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:54:34.386778 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.386745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb\" (UID: \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:54:34.389484 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.389451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb\" (UID: \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:54:34.441872 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.441842    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"]
Jul 29 19:54:34.445577 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.445561    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"
Jul 29 19:54:34.448752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.448730    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9d9f53\""
Jul 29 19:54:34.456579 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.456552    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"]
Jul 29 19:54:34.467925 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.467894    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"]
Jul 29 19:54:34.471611 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.471591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"
Jul 29 19:54:34.474628 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.474608    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9d9f53-dockercfg-7zwrd\""
Jul 29 19:54:34.480293 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.480268    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"]
Jul 29 19:54:34.487779 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.487751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195f9ec6-3a39-4ea4-864c-4a680c23e176-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj\" (UID: \"195f9ec6-3a39-4ea4-864c-4a680c23e176\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"
Jul 29 19:54:34.487923 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.487798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k5lc\" (UniqueName: \"kubernetes.io/projected/195f9ec6-3a39-4ea4-864c-4a680c23e176-kube-api-access-6k5lc\") pod \"payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj\" (UID: \"195f9ec6-3a39-4ea4-864c-4a680c23e176\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"
Jul 29 19:54:34.487923 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.487886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpdvx\" (UniqueName: \"kubernetes.io/projected/bbe4f538-6c3d-458b-b120-7ac00b311a39-kube-api-access-rpdvx\") pod \"payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh\" (UID: \"bbe4f538-6c3d-458b-b120-7ac00b311a39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"
Jul 29 19:54:34.488015 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.487992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe4f538-6c3d-458b-b120-7ac00b311a39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh\" (UID: \"bbe4f538-6c3d-458b-b120-7ac00b311a39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"
Jul 29 19:54:34.589331 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.589297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpdvx\" (UniqueName: \"kubernetes.io/projected/bbe4f538-6c3d-458b-b120-7ac00b311a39-kube-api-access-rpdvx\") pod \"payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh\" (UID: \"bbe4f538-6c3d-458b-b120-7ac00b311a39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"
Jul 29 19:54:34.589437 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.589409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe4f538-6c3d-458b-b120-7ac00b311a39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh\" (UID: \"bbe4f538-6c3d-458b-b120-7ac00b311a39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"
Jul 29 19:54:34.589506 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.589492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195f9ec6-3a39-4ea4-864c-4a680c23e176-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj\" (UID: \"195f9ec6-3a39-4ea4-864c-4a680c23e176\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"
Jul 29 19:54:34.589544 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.589526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k5lc\" (UniqueName: \"kubernetes.io/projected/195f9ec6-3a39-4ea4-864c-4a680c23e176-kube-api-access-6k5lc\") pod \"payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj\" (UID: \"195f9ec6-3a39-4ea4-864c-4a680c23e176\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"
Jul 29 19:54:34.590360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.590339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195f9ec6-3a39-4ea4-864c-4a680c23e176-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj\" (UID: \"195f9ec6-3a39-4ea4-864c-4a680c23e176\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"
Jul 29 19:54:34.590512 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.590497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe4f538-6c3d-458b-b120-7ac00b311a39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh\" (UID: \"bbe4f538-6c3d-458b-b120-7ac00b311a39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"
Jul 29 19:54:34.598040 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.598013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k5lc\" (UniqueName: \"kubernetes.io/projected/195f9ec6-3a39-4ea4-864c-4a680c23e176-kube-api-access-6k5lc\") pod \"payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj\" (UID: \"195f9ec6-3a39-4ea4-864c-4a680c23e176\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"
Jul 29 19:54:34.598354 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.598337    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpdvx\" (UniqueName: \"kubernetes.io/projected/bbe4f538-6c3d-458b-b120-7ac00b311a39-kube-api-access-rpdvx\") pod \"payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh\" (UID: \"bbe4f538-6c3d-458b-b120-7ac00b311a39\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"
Jul 29 19:54:34.666343 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.666262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:54:34.758499 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.758467    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"
Jul 29 19:54:34.784052 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.783849    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"
Jul 29 19:54:34.832444 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.830953    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"]
Jul 29 19:54:34.842570 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.842537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb" event={"ID":"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31","Type":"ContainerStarted","Data":"49545120c6f3c15e865e4d8085ffadf1d227725325e59b29e41b5aacf6b194e7"}
Jul 29 19:54:34.928480 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.928447    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"]
Jul 29 19:54:34.931023 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:54:34.930953    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe4f538_6c3d_458b_b120_7ac00b311a39.slice/crio-4a68842cac5eebc4821a1f26cd7b68c6fef391dcbbe16e7fa60b69ad61c2513d WatchSource:0}: Error finding container 4a68842cac5eebc4821a1f26cd7b68c6fef391dcbbe16e7fa60b69ad61c2513d: Status 404 returned error can't find the container with id 4a68842cac5eebc4821a1f26cd7b68c6fef391dcbbe16e7fa60b69ad61c2513d
Jul 29 19:54:34.964100 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:34.963743    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"]
Jul 29 19:54:34.966752 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:54:34.966703    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195f9ec6_3a39_4ea4_864c_4a680c23e176.slice/crio-4f32aec16cf94085b2c3e6a406ef133eb09cfeef2484303caecaedf0aace6612 WatchSource:0}: Error finding container 4f32aec16cf94085b2c3e6a406ef133eb09cfeef2484303caecaedf0aace6612: Status 404 returned error can't find the container with id 4f32aec16cf94085b2c3e6a406ef133eb09cfeef2484303caecaedf0aace6612
Jul 29 19:54:35.849654 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:35.849615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh" event={"ID":"bbe4f538-6c3d-458b-b120-7ac00b311a39","Type":"ContainerStarted","Data":"91122e8d7ebda8ace26b82309ebe3ef38e98abce1e24900f90c0c803e87007b6"}
Jul 29 19:54:35.850080 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:35.849676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh" event={"ID":"bbe4f538-6c3d-458b-b120-7ac00b311a39","Type":"ContainerStarted","Data":"4a68842cac5eebc4821a1f26cd7b68c6fef391dcbbe16e7fa60b69ad61c2513d"}
Jul 29 19:54:35.850080 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:35.849798    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"
Jul 29 19:54:35.851152 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:35.851129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb" event={"ID":"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31","Type":"ContainerStarted","Data":"d57361a030485ba41a4aae149494a9167bb1384b63a21edd0bcac81985d87dee"}
Jul 29 19:54:35.851298 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:35.851207    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:54:35.852652 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:35.852632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj" event={"ID":"195f9ec6-3a39-4ea4-864c-4a680c23e176","Type":"ContainerStarted","Data":"aded5ad1c7e0da6b0af6c830a569da64ae16df28bf80b3f1e85aa26f6471d5a3"}
Jul 29 19:54:35.852749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:35.852667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj" event={"ID":"195f9ec6-3a39-4ea4-864c-4a680c23e176","Type":"ContainerStarted","Data":"4f32aec16cf94085b2c3e6a406ef133eb09cfeef2484303caecaedf0aace6612"}
Jul 29 19:54:35.852749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:35.852687    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"
Jul 29 19:54:35.867499 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:35.867453    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh" podStartSLOduration=1.867440419 podStartE2EDuration="1.867440419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:35.865308932 +0000 UTC m=+3242.164981454" watchObservedRunningTime="2026-07-29 19:54:35.867440419 +0000 UTC m=+3242.167112957"
Jul 29 19:54:35.883578 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:35.883529    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb" podStartSLOduration=2.381969629 podStartE2EDuration="2.883515876s" totalImagesPullingTime="501.546247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:35.883446135 +0000 UTC m=+3242.183118653" watchObservedRunningTime="2026-07-29 19:54:35.883515876 +0000 UTC m=+3242.183188377"
Jul 29 19:54:35.902936 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:35.902399    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj" podStartSLOduration=1.9023813029999999 podStartE2EDuration="1.902381303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:35.901407181 +0000 UTC m=+3242.201079683" watchObservedRunningTime="2026-07-29 19:54:35.902381303 +0000 UTC m=+3242.202053806"
Jul 29 19:54:41.863618 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:41.863583    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:54:46.860494 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:46.860466    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"
Jul 29 19:54:46.860983 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:54:46.860701    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"
Jul 29 19:55:02.714915 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:02.714872    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"]
Jul 29 19:55:02.733558 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:02.733522    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"]
Jul 29 19:55:02.733766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:02.733676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:55:02.736515 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:02.736488    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f36365-serving-cert\""
Jul 29 19:55:02.868849 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:02.868803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e109d8-682c-4c9a-85c0-b4f177eee884-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n\" (UID: \"06e109d8-682c-4c9a-85c0-b4f177eee884\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:55:02.868849 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:02.868845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e109d8-682c-4c9a-85c0-b4f177eee884-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n\" (UID: \"06e109d8-682c-4c9a-85c0-b4f177eee884\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:55:02.869086 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:02.868868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/06e109d8-682c-4c9a-85c0-b4f177eee884-kube-api-access-m5pgv\") pod \"maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n\" (UID: \"06e109d8-682c-4c9a-85c0-b4f177eee884\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:55:02.969690 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:02.969597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/06e109d8-682c-4c9a-85c0-b4f177eee884-kube-api-access-m5pgv\") pod \"maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n\" (UID: \"06e109d8-682c-4c9a-85c0-b4f177eee884\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:55:02.969874 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:02.969715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e109d8-682c-4c9a-85c0-b4f177eee884-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n\" (UID: \"06e109d8-682c-4c9a-85c0-b4f177eee884\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:55:02.969874 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:02.969735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e109d8-682c-4c9a-85c0-b4f177eee884-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n\" (UID: \"06e109d8-682c-4c9a-85c0-b4f177eee884\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:55:02.970392 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:02.970371    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e109d8-682c-4c9a-85c0-b4f177eee884-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n\" (UID: \"06e109d8-682c-4c9a-85c0-b4f177eee884\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:55:02.972493 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:02.972472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e109d8-682c-4c9a-85c0-b4f177eee884-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n\" (UID: \"06e109d8-682c-4c9a-85c0-b4f177eee884\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:55:02.979459 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:02.979432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/06e109d8-682c-4c9a-85c0-b4f177eee884-kube-api-access-m5pgv\") pod \"maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n\" (UID: \"06e109d8-682c-4c9a-85c0-b4f177eee884\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:55:03.048141 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.048099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:55:03.201174 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.201138    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"]
Jul 29 19:55:03.202723 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:55:03.202697    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e109d8_682c_4c9a_85c0_b4f177eee884.slice/crio-ac93082999e9367fbf263793a2e159071f472681b6600fbba85abd935fa82dff WatchSource:0}: Error finding container ac93082999e9367fbf263793a2e159071f472681b6600fbba85abd935fa82dff: Status 404 returned error can't find the container with id ac93082999e9367fbf263793a2e159071f472681b6600fbba85abd935fa82dff
Jul 29 19:55:03.401077 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.401040    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"]
Jul 29 19:55:03.405890 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.405867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"
Jul 29 19:55:03.408977 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.408957    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f36365\""
Jul 29 19:55:03.409117 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.409097    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f36365-dockercfg-qd496\""
Jul 29 19:55:03.419881 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.419846    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"]
Jul 29 19:55:03.437045 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.436989    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"]
Jul 29 19:55:03.441299 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.441270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"
Jul 29 19:55:03.449633 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.449601    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"]
Jul 29 19:55:03.576664 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.576631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2\" (UID: \"7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"
Jul 29 19:55:03.576806 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.576711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtq8n\" (UniqueName: \"kubernetes.io/projected/689f313a-0533-4967-a237-08406df3b45b-kube-api-access-rtq8n\") pod \"payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47\" (UID: \"689f313a-0533-4967-a237-08406df3b45b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"
Jul 29 19:55:03.576806 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.576798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkl7c\" (UniqueName: \"kubernetes.io/projected/7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3-kube-api-access-wkl7c\") pod \"payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2\" (UID: \"7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"
Jul 29 19:55:03.576886 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.576860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689f313a-0533-4967-a237-08406df3b45b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47\" (UID: \"689f313a-0533-4967-a237-08406df3b45b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"
Jul 29 19:55:03.677812 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.677711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2\" (UID: \"7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"
Jul 29 19:55:03.677812 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.677768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtq8n\" (UniqueName: \"kubernetes.io/projected/689f313a-0533-4967-a237-08406df3b45b-kube-api-access-rtq8n\") pod \"payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47\" (UID: \"689f313a-0533-4967-a237-08406df3b45b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"
Jul 29 19:55:03.677812 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.677816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkl7c\" (UniqueName: \"kubernetes.io/projected/7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3-kube-api-access-wkl7c\") pod \"payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2\" (UID: \"7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"
Jul 29 19:55:03.678102 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.677832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689f313a-0533-4967-a237-08406df3b45b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47\" (UID: \"689f313a-0533-4967-a237-08406df3b45b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"
Jul 29 19:55:03.678602 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.678582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689f313a-0533-4967-a237-08406df3b45b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47\" (UID: \"689f313a-0533-4967-a237-08406df3b45b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"
Jul 29 19:55:03.678690 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.678667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2\" (UID: \"7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"
Jul 29 19:55:03.687661 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.687628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkl7c\" (UniqueName: \"kubernetes.io/projected/7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3-kube-api-access-wkl7c\") pod \"payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2\" (UID: \"7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"
Jul 29 19:55:03.687802 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.687666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtq8n\" (UniqueName: \"kubernetes.io/projected/689f313a-0533-4967-a237-08406df3b45b-kube-api-access-rtq8n\") pod \"payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47\" (UID: \"689f313a-0533-4967-a237-08406df3b45b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"
Jul 29 19:55:03.718347 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.718312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"
Jul 29 19:55:03.755024 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.754972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"
Jul 29 19:55:03.883808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.883758    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"]
Jul 29 19:55:03.885324 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:55:03.885279    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c48f31a_f6d7_4672_a8e5_1c2e47c1d4d3.slice/crio-5574c41c598f868e210305330d5b820670a7450f29dbc8b4910d3be51f88a319 WatchSource:0}: Error finding container 5574c41c598f868e210305330d5b820670a7450f29dbc8b4910d3be51f88a319: Status 404 returned error can't find the container with id 5574c41c598f868e210305330d5b820670a7450f29dbc8b4910d3be51f88a319
Jul 29 19:55:03.942813 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:03.942780    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"]
Jul 29 19:55:03.944022 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:55:03.943977    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689f313a_0533_4967_a237_08406df3b45b.slice/crio-35c67263920a395dbc7c200d97251c527c8b562b675fbfb267a944cf196841b6 WatchSource:0}: Error finding container 35c67263920a395dbc7c200d97251c527c8b562b675fbfb267a944cf196841b6: Status 404 returned error can't find the container with id 35c67263920a395dbc7c200d97251c527c8b562b675fbfb267a944cf196841b6
Jul 29 19:55:04.010271 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:04.010228    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2" event={"ID":"7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3","Type":"ContainerStarted","Data":"a466e497ab2500a554d7f0da808775937d1b85ab2608764e1ee2887fde80938e"}
Jul 29 19:55:04.010489 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:04.010294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2" event={"ID":"7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3","Type":"ContainerStarted","Data":"5574c41c598f868e210305330d5b820670a7450f29dbc8b4910d3be51f88a319"}
Jul 29 19:55:04.010489 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:04.010336    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"
Jul 29 19:55:04.011600 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:04.011574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47" event={"ID":"689f313a-0533-4967-a237-08406df3b45b","Type":"ContainerStarted","Data":"35c67263920a395dbc7c200d97251c527c8b562b675fbfb267a944cf196841b6"}
Jul 29 19:55:04.012998 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:04.012977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n" event={"ID":"06e109d8-682c-4c9a-85c0-b4f177eee884","Type":"ContainerStarted","Data":"bc55dcaa970e9fc0551b673489cf47cf45799c82c4e5cbaaba1cfd10b2713f7c"}
Jul 29 19:55:04.013109 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:04.013013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n" event={"ID":"06e109d8-682c-4c9a-85c0-b4f177eee884","Type":"ContainerStarted","Data":"ac93082999e9367fbf263793a2e159071f472681b6600fbba85abd935fa82dff"}
Jul 29 19:55:04.013153 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:04.013121    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:55:04.027051 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:04.026979    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2" podStartSLOduration=1.026960343 podStartE2EDuration="1.026960343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:55:04.026044205 +0000 UTC m=+3270.325716718" watchObservedRunningTime="2026-07-29 19:55:04.026960343 +0000 UTC m=+3270.326632844"
Jul 29 19:55:04.043551 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:04.043503    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n" podStartSLOduration=1.6898148370000001 podStartE2EDuration="2.04348479s" totalImagesPullingTime="353.669953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:55:04.042882881 +0000 UTC m=+3270.342555459" watchObservedRunningTime="2026-07-29 19:55:04.04348479 +0000 UTC m=+3270.343157290"
Jul 29 19:55:05.020115 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:05.020074    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47" event={"ID":"689f313a-0533-4967-a237-08406df3b45b","Type":"ContainerStarted","Data":"91ea4f6ff39a016af2b08ac930d3172c7cf40150e2c587afda9bfd07d1850bf5"}
Jul 29 19:55:05.020583 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:05.020142    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"
Jul 29 19:55:05.057350 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:05.057296    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47" podStartSLOduration=2.057277092 podStartE2EDuration="2.057277092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:55:05.055265175 +0000 UTC m=+3271.354937678" watchObservedRunningTime="2026-07-29 19:55:05.057277092 +0000 UTC m=+3271.356949591"
Jul 29 19:55:10.025591 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:10.025553    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:55:15.023443 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:15.023405    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"
Jul 29 19:55:16.035597 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:16.035567    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"
Jul 29 19:55:34.402859 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:34.402827    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:55:34.412214 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:34.412159    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:55:34.499398 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:34.499363    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dhqmz_af11b487-3ae7-4c80-b7d1-1a54e4593381/payload-processing/0.log"
Jul 29 19:55:34.624498 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:55:34.624468    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47_689f313a-0533-4967-a237-08406df3b45b/payload-processing/0.log"
Jul 29 19:56:21.494850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:21.494817    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj_195f9ec6-3a39-4ea4-864c-4a680c23e176/payload-processing/0.log"
Jul 29 19:56:21.614616 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:21.614589    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dhqmz_af11b487-3ae7-4c80-b7d1-1a54e4593381/payload-processing/0.log"
Jul 29 19:56:21.730174 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:21.730145    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47_689f313a-0533-4967-a237-08406df3b45b/payload-processing/0.log"
Jul 29 19:56:25.401724 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.401691    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"]
Jul 29 19:56:25.405719 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.405696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:25.409004 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.408971    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-040fde-serving-cert\""
Jul 29 19:56:25.410530 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.410503    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"]
Jul 29 19:56:25.538369 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.538331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee2996c-86dd-405f-a049-c909121ee879-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn\" (UID: \"0ee2996c-86dd-405f-a049-c909121ee879\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:25.538572 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.538386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ee2996c-86dd-405f-a049-c909121ee879-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn\" (UID: \"0ee2996c-86dd-405f-a049-c909121ee879\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:25.538625 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.538564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6dgd\" (UniqueName: \"kubernetes.io/projected/0ee2996c-86dd-405f-a049-c909121ee879-kube-api-access-m6dgd\") pod \"maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn\" (UID: \"0ee2996c-86dd-405f-a049-c909121ee879\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:25.639607 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.639566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6dgd\" (UniqueName: \"kubernetes.io/projected/0ee2996c-86dd-405f-a049-c909121ee879-kube-api-access-m6dgd\") pod \"maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn\" (UID: \"0ee2996c-86dd-405f-a049-c909121ee879\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:25.639799 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.639638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee2996c-86dd-405f-a049-c909121ee879-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn\" (UID: \"0ee2996c-86dd-405f-a049-c909121ee879\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:25.639799 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.639671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ee2996c-86dd-405f-a049-c909121ee879-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn\" (UID: \"0ee2996c-86dd-405f-a049-c909121ee879\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:25.640339 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.640318    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee2996c-86dd-405f-a049-c909121ee879-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn\" (UID: \"0ee2996c-86dd-405f-a049-c909121ee879\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:25.642339 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.642319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ee2996c-86dd-405f-a049-c909121ee879-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn\" (UID: \"0ee2996c-86dd-405f-a049-c909121ee879\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:25.653587 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.653523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6dgd\" (UniqueName: \"kubernetes.io/projected/0ee2996c-86dd-405f-a049-c909121ee879-kube-api-access-m6dgd\") pod \"maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn\" (UID: \"0ee2996c-86dd-405f-a049-c909121ee879\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:25.718155 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.718125    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:25.866055 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:25.866026    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"]
Jul 29 19:56:25.867716 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:56:25.867689    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee2996c_86dd_405f_a049_c909121ee879.slice/crio-83c58452c6bbc608a3b1213936280a0dc6913e49f87e1b448cec92574a271d49 WatchSource:0}: Error finding container 83c58452c6bbc608a3b1213936280a0dc6913e49f87e1b448cec92574a271d49: Status 404 returned error can't find the container with id 83c58452c6bbc608a3b1213936280a0dc6913e49f87e1b448cec92574a271d49
Jul 29 19:56:26.125946 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.125864    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"]
Jul 29 19:56:26.130823 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.130800    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"
Jul 29 19:56:26.133769 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.133746    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-040fde\""
Jul 29 19:56:26.133897 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.133779    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-040fde-dockercfg-km5wl\""
Jul 29 19:56:26.140051 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.140025    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"]
Jul 29 19:56:26.173590 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.173545    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"]
Jul 29 19:56:26.177357 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.177329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"
Jul 29 19:56:26.189750 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.189722    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"]
Jul 29 19:56:26.245701 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.245667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54adcb4d-56f8-4912-a129-eff2763aa204-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7\" (UID: \"54adcb4d-56f8-4912-a129-eff2763aa204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"
Jul 29 19:56:26.245890 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.245715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmw7\" (UniqueName: \"kubernetes.io/projected/bbf28d22-bb2e-4151-8f9c-44eb0fc714b7-kube-api-access-jvmw7\") pod \"payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg\" (UID: \"bbf28d22-bb2e-4151-8f9c-44eb0fc714b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"
Jul 29 19:56:26.245890 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.245762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj72h\" (UniqueName: \"kubernetes.io/projected/54adcb4d-56f8-4912-a129-eff2763aa204-kube-api-access-nj72h\") pod \"payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7\" (UID: \"54adcb4d-56f8-4912-a129-eff2763aa204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"
Jul 29 19:56:26.245890 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.245816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf28d22-bb2e-4151-8f9c-44eb0fc714b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg\" (UID: \"bbf28d22-bb2e-4151-8f9c-44eb0fc714b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"
Jul 29 19:56:26.346970 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.346924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54adcb4d-56f8-4912-a129-eff2763aa204-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7\" (UID: \"54adcb4d-56f8-4912-a129-eff2763aa204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"
Jul 29 19:56:26.346970 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.346976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmw7\" (UniqueName: \"kubernetes.io/projected/bbf28d22-bb2e-4151-8f9c-44eb0fc714b7-kube-api-access-jvmw7\") pod \"payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg\" (UID: \"bbf28d22-bb2e-4151-8f9c-44eb0fc714b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"
Jul 29 19:56:26.347281 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.346995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj72h\" (UniqueName: \"kubernetes.io/projected/54adcb4d-56f8-4912-a129-eff2763aa204-kube-api-access-nj72h\") pod \"payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7\" (UID: \"54adcb4d-56f8-4912-a129-eff2763aa204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"
Jul 29 19:56:26.347281 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.347025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf28d22-bb2e-4151-8f9c-44eb0fc714b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg\" (UID: \"bbf28d22-bb2e-4151-8f9c-44eb0fc714b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"
Jul 29 19:56:26.347804 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.347779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54adcb4d-56f8-4912-a129-eff2763aa204-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7\" (UID: \"54adcb4d-56f8-4912-a129-eff2763aa204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"
Jul 29 19:56:26.347893 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.347779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf28d22-bb2e-4151-8f9c-44eb0fc714b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg\" (UID: \"bbf28d22-bb2e-4151-8f9c-44eb0fc714b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"
Jul 29 19:56:26.356738 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.356714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmw7\" (UniqueName: \"kubernetes.io/projected/bbf28d22-bb2e-4151-8f9c-44eb0fc714b7-kube-api-access-jvmw7\") pod \"payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg\" (UID: \"bbf28d22-bb2e-4151-8f9c-44eb0fc714b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"
Jul 29 19:56:26.356829 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.356755    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj72h\" (UniqueName: \"kubernetes.io/projected/54adcb4d-56f8-4912-a129-eff2763aa204-kube-api-access-nj72h\") pod \"payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7\" (UID: \"54adcb4d-56f8-4912-a129-eff2763aa204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"
Jul 29 19:56:26.444119 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.444090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"
Jul 29 19:56:26.457748 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.457721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn" event={"ID":"0ee2996c-86dd-405f-a049-c909121ee879","Type":"ContainerStarted","Data":"bb9f0ea1e7715a3039e0b5ffed1b134a7f3333c7993ce31e073c3a2371db4437"}
Jul 29 19:56:26.457873 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.457773    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn" event={"ID":"0ee2996c-86dd-405f-a049-c909121ee879","Type":"ContainerStarted","Data":"83c58452c6bbc608a3b1213936280a0dc6913e49f87e1b448cec92574a271d49"}
Jul 29 19:56:26.457873 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.457820    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:26.475234 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.475159    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn" podStartSLOduration=1.084806891 podStartE2EDuration="1.475137462s" totalImagesPullingTime="390.330571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:56:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:56:26.473813164 +0000 UTC m=+3352.773485675" watchObservedRunningTime="2026-07-29 19:56:26.475137462 +0000 UTC m=+3352.774809962"
Jul 29 19:56:26.488937 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.488896    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"
Jul 29 19:56:26.599372 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.599340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"]
Jul 29 19:56:26.601790 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:56:26.601765    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54adcb4d_56f8_4912_a129_eff2763aa204.slice/crio-01b3b5f9c20ff6c23ea878a9fe89c71d1eeb8d248377fdf61c2877264c798b8d WatchSource:0}: Error finding container 01b3b5f9c20ff6c23ea878a9fe89c71d1eeb8d248377fdf61c2877264c798b8d: Status 404 returned error can't find the container with id 01b3b5f9c20ff6c23ea878a9fe89c71d1eeb8d248377fdf61c2877264c798b8d
Jul 29 19:56:26.651458 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:26.651426    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"]
Jul 29 19:56:26.652705 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:56:26.652676    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf28d22_bb2e_4151_8f9c_44eb0fc714b7.slice/crio-76ed9f8b5a4baddd6e0a690d1f01e56429fd569c03a244070d89a0ff04c8722e WatchSource:0}: Error finding container 76ed9f8b5a4baddd6e0a690d1f01e56429fd569c03a244070d89a0ff04c8722e: Status 404 returned error can't find the container with id 76ed9f8b5a4baddd6e0a690d1f01e56429fd569c03a244070d89a0ff04c8722e
Jul 29 19:56:27.469860 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:27.469803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg" event={"ID":"bbf28d22-bb2e-4151-8f9c-44eb0fc714b7","Type":"ContainerStarted","Data":"f7bd3b04423a2431471bd692f08941e6d9c20b32fd25c16f9cbc1b5ed4d230b6"}
Jul 29 19:56:27.470455 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:27.469904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg" event={"ID":"bbf28d22-bb2e-4151-8f9c-44eb0fc714b7","Type":"ContainerStarted","Data":"76ed9f8b5a4baddd6e0a690d1f01e56429fd569c03a244070d89a0ff04c8722e"}
Jul 29 19:56:27.470455 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:27.469932    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"
Jul 29 19:56:27.471791 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:27.471764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7" event={"ID":"54adcb4d-56f8-4912-a129-eff2763aa204","Type":"ContainerStarted","Data":"ae6f548ae4112ea8d47998c67834c480598c4a605d79778b817a38a233fef676"}
Jul 29 19:56:27.471910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:27.471810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7" event={"ID":"54adcb4d-56f8-4912-a129-eff2763aa204","Type":"ContainerStarted","Data":"01b3b5f9c20ff6c23ea878a9fe89c71d1eeb8d248377fdf61c2877264c798b8d"}
Jul 29 19:56:27.472017 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:27.471999    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"
Jul 29 19:56:27.488431 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:27.488384    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg" podStartSLOduration=1.488370356 podStartE2EDuration="1.488370356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:56:27.485227887 +0000 UTC m=+3353.784900385" watchObservedRunningTime="2026-07-29 19:56:27.488370356 +0000 UTC m=+3353.788042855"
Jul 29 19:56:27.502420 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:27.502366    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7" podStartSLOduration=1.502352584 podStartE2EDuration="1.502352584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:56:27.500309441 +0000 UTC m=+3353.799981942" watchObservedRunningTime="2026-07-29 19:56:27.502352584 +0000 UTC m=+3353.802025082"
Jul 29 19:56:32.477667 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:32.477638    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:38.478865 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:38.478834    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"
Jul 29 19:56:38.479366 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:38.478908    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"
Jul 29 19:56:55.869774 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:55.869736    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"]
Jul 29 19:56:55.870319 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:55.869980    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn" podUID="0ee2996c-86dd-405f-a049-c909121ee879" containerName="maas-api" containerID="cri-o://bb9f0ea1e7715a3039e0b5ffed1b134a7f3333c7993ce31e073c3a2371db4437" gracePeriod=30
Jul 29 19:56:55.922715 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:55.922678    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"]
Jul 29 19:56:55.923273 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:55.922922    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg" podUID="bbf28d22-bb2e-4151-8f9c-44eb0fc714b7" containerName="payload-processing" containerID="cri-o://f7bd3b04423a2431471bd692f08941e6d9c20b32fd25c16f9cbc1b5ed4d230b6" gracePeriod=30
Jul 29 19:56:55.929406 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:55.929374    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"]
Jul 29 19:56:55.929627 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:55.929603    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7" podUID="54adcb4d-56f8-4912-a129-eff2763aa204" containerName="payload-pre-processing" containerID="cri-o://ae6f548ae4112ea8d47998c67834c480598c4a605d79778b817a38a233fef676" gracePeriod=30
Jul 29 19:56:56.239766 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.239407    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:56.256528 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.256496    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"
Jul 29 19:56:56.262070 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.262041    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"
Jul 29 19:56:56.342604 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.342546    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee2996c-86dd-405f-a049-c909121ee879-openshift-service-ca\") pod \"0ee2996c-86dd-405f-a049-c909121ee879\" (UID: \"0ee2996c-86dd-405f-a049-c909121ee879\") "
Jul 29 19:56:56.342604 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.342610    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54adcb4d-56f8-4912-a129-eff2763aa204-plugins-config-volume\") pod \"54adcb4d-56f8-4912-a129-eff2763aa204\" (UID: \"54adcb4d-56f8-4912-a129-eff2763aa204\") "
Jul 29 19:56:56.342882 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.342661    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf28d22-bb2e-4151-8f9c-44eb0fc714b7-plugins-config-volume\") pod \"bbf28d22-bb2e-4151-8f9c-44eb0fc714b7\" (UID: \"bbf28d22-bb2e-4151-8f9c-44eb0fc714b7\") "
Jul 29 19:56:56.342882 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.342716    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6dgd\" (UniqueName: \"kubernetes.io/projected/0ee2996c-86dd-405f-a049-c909121ee879-kube-api-access-m6dgd\") pod \"0ee2996c-86dd-405f-a049-c909121ee879\" (UID: \"0ee2996c-86dd-405f-a049-c909121ee879\") "
Jul 29 19:56:56.342882 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.342758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ee2996c-86dd-405f-a049-c909121ee879-maas-api-tls\") pod \"0ee2996c-86dd-405f-a049-c909121ee879\" (UID: \"0ee2996c-86dd-405f-a049-c909121ee879\") "
Jul 29 19:56:56.342882 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.342800    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj72h\" (UniqueName: \"kubernetes.io/projected/54adcb4d-56f8-4912-a129-eff2763aa204-kube-api-access-nj72h\") pod \"54adcb4d-56f8-4912-a129-eff2763aa204\" (UID: \"54adcb4d-56f8-4912-a129-eff2763aa204\") "
Jul 29 19:56:56.342882 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.342870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvmw7\" (UniqueName: \"kubernetes.io/projected/bbf28d22-bb2e-4151-8f9c-44eb0fc714b7-kube-api-access-jvmw7\") pod \"bbf28d22-bb2e-4151-8f9c-44eb0fc714b7\" (UID: \"bbf28d22-bb2e-4151-8f9c-44eb0fc714b7\") "
Jul 29 19:56:56.343142 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.343071    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee2996c-86dd-405f-a049-c909121ee879-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ee2996c-86dd-405f-a049-c909121ee879" (UID: "0ee2996c-86dd-405f-a049-c909121ee879"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:56:56.343142 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.343097    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54adcb4d-56f8-4912-a129-eff2763aa204-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54adcb4d-56f8-4912-a129-eff2763aa204" (UID: "54adcb4d-56f8-4912-a129-eff2763aa204"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:56:56.343142 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.343101    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf28d22-bb2e-4151-8f9c-44eb0fc714b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbf28d22-bb2e-4151-8f9c-44eb0fc714b7" (UID: "bbf28d22-bb2e-4151-8f9c-44eb0fc714b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:56:56.343364 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.343337    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee2996c-86dd-405f-a049-c909121ee879-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:56:56.343364 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.343359    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54adcb4d-56f8-4912-a129-eff2763aa204-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:56:56.343483 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.343375    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf28d22-bb2e-4151-8f9c-44eb0fc714b7-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:56:56.345209 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.345162    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee2996c-86dd-405f-a049-c909121ee879-kube-api-access-m6dgd" (OuterVolumeSpecName: "kube-api-access-m6dgd") pod "0ee2996c-86dd-405f-a049-c909121ee879" (UID: "0ee2996c-86dd-405f-a049-c909121ee879"). InnerVolumeSpecName "kube-api-access-m6dgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:56:56.345209 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.345166    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ee2996c-86dd-405f-a049-c909121ee879-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ee2996c-86dd-405f-a049-c909121ee879" (UID: "0ee2996c-86dd-405f-a049-c909121ee879"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:56:56.345387 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.345187    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54adcb4d-56f8-4912-a129-eff2763aa204-kube-api-access-nj72h" (OuterVolumeSpecName: "kube-api-access-nj72h") pod "54adcb4d-56f8-4912-a129-eff2763aa204" (UID: "54adcb4d-56f8-4912-a129-eff2763aa204"). InnerVolumeSpecName "kube-api-access-nj72h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:56:56.345387 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.345255    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf28d22-bb2e-4151-8f9c-44eb0fc714b7-kube-api-access-jvmw7" (OuterVolumeSpecName: "kube-api-access-jvmw7") pod "bbf28d22-bb2e-4151-8f9c-44eb0fc714b7" (UID: "bbf28d22-bb2e-4151-8f9c-44eb0fc714b7"). InnerVolumeSpecName "kube-api-access-jvmw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:56:56.444027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.443908    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvmw7\" (UniqueName: \"kubernetes.io/projected/bbf28d22-bb2e-4151-8f9c-44eb0fc714b7-kube-api-access-jvmw7\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:56:56.444027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.443962    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6dgd\" (UniqueName: \"kubernetes.io/projected/0ee2996c-86dd-405f-a049-c909121ee879-kube-api-access-m6dgd\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:56:56.444027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.443972    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ee2996c-86dd-405f-a049-c909121ee879-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:56:56.444027 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.443981    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj72h\" (UniqueName: \"kubernetes.io/projected/54adcb4d-56f8-4912-a129-eff2763aa204-kube-api-access-nj72h\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:56:56.646633 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.646593    2579 generic.go:356] "Generic (PLEG): container finished" podID="bbf28d22-bb2e-4151-8f9c-44eb0fc714b7" containerID="f7bd3b04423a2431471bd692f08941e6d9c20b32fd25c16f9cbc1b5ed4d230b6" exitCode=0
Jul 29 19:56:56.646826 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.646641    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"
Jul 29 19:56:56.646826 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.646685    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg" event={"ID":"bbf28d22-bb2e-4151-8f9c-44eb0fc714b7","Type":"ContainerDied","Data":"f7bd3b04423a2431471bd692f08941e6d9c20b32fd25c16f9cbc1b5ed4d230b6"}
Jul 29 19:56:56.646826 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.646737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg" event={"ID":"bbf28d22-bb2e-4151-8f9c-44eb0fc714b7","Type":"ContainerDied","Data":"76ed9f8b5a4baddd6e0a690d1f01e56429fd569c03a244070d89a0ff04c8722e"}
Jul 29 19:56:56.646826 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.646769    2579 scope.go:117] "RemoveContainer" containerID="f7bd3b04423a2431471bd692f08941e6d9c20b32fd25c16f9cbc1b5ed4d230b6"
Jul 29 19:56:56.648300 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.648270    2579 generic.go:356] "Generic (PLEG): container finished" podID="54adcb4d-56f8-4912-a129-eff2763aa204" containerID="ae6f548ae4112ea8d47998c67834c480598c4a605d79778b817a38a233fef676" exitCode=0
Jul 29 19:56:56.648434 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.648315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7" event={"ID":"54adcb4d-56f8-4912-a129-eff2763aa204","Type":"ContainerDied","Data":"ae6f548ae4112ea8d47998c67834c480598c4a605d79778b817a38a233fef676"}
Jul 29 19:56:56.648434 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.648372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7" event={"ID":"54adcb4d-56f8-4912-a129-eff2763aa204","Type":"ContainerDied","Data":"01b3b5f9c20ff6c23ea878a9fe89c71d1eeb8d248377fdf61c2877264c798b8d"}
Jul 29 19:56:56.648559 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.648456    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"
Jul 29 19:56:56.649990 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.649950    2579 generic.go:356] "Generic (PLEG): container finished" podID="0ee2996c-86dd-405f-a049-c909121ee879" containerID="bb9f0ea1e7715a3039e0b5ffed1b134a7f3333c7993ce31e073c3a2371db4437" exitCode=0
Jul 29 19:56:56.649990 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.649983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn" event={"ID":"0ee2996c-86dd-405f-a049-c909121ee879","Type":"ContainerDied","Data":"bb9f0ea1e7715a3039e0b5ffed1b134a7f3333c7993ce31e073c3a2371db4437"}
Jul 29 19:56:56.650140 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.650018    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"
Jul 29 19:56:56.650140 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.650033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn" event={"ID":"0ee2996c-86dd-405f-a049-c909121ee879","Type":"ContainerDied","Data":"83c58452c6bbc608a3b1213936280a0dc6913e49f87e1b448cec92574a271d49"}
Jul 29 19:56:56.658180 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.658155    2579 scope.go:117] "RemoveContainer" containerID="f7bd3b04423a2431471bd692f08941e6d9c20b32fd25c16f9cbc1b5ed4d230b6"
Jul 29 19:56:56.658559 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:56:56.658538    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7bd3b04423a2431471bd692f08941e6d9c20b32fd25c16f9cbc1b5ed4d230b6\": container with ID starting with f7bd3b04423a2431471bd692f08941e6d9c20b32fd25c16f9cbc1b5ed4d230b6 not found: ID does not exist" containerID="f7bd3b04423a2431471bd692f08941e6d9c20b32fd25c16f9cbc1b5ed4d230b6"
Jul 29 19:56:56.658648 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.658567    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bd3b04423a2431471bd692f08941e6d9c20b32fd25c16f9cbc1b5ed4d230b6"} err="failed to get container status \"f7bd3b04423a2431471bd692f08941e6d9c20b32fd25c16f9cbc1b5ed4d230b6\": rpc error: code = NotFound desc = could not find container \"f7bd3b04423a2431471bd692f08941e6d9c20b32fd25c16f9cbc1b5ed4d230b6\": container with ID starting with f7bd3b04423a2431471bd692f08941e6d9c20b32fd25c16f9cbc1b5ed4d230b6 not found: ID does not exist"
Jul 29 19:56:56.658648 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.658585    2579 scope.go:117] "RemoveContainer" containerID="ae6f548ae4112ea8d47998c67834c480598c4a605d79778b817a38a233fef676"
Jul 29 19:56:56.668370 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.668346    2579 scope.go:117] "RemoveContainer" containerID="ae6f548ae4112ea8d47998c67834c480598c4a605d79778b817a38a233fef676"
Jul 29 19:56:56.668667 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:56:56.668648    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6f548ae4112ea8d47998c67834c480598c4a605d79778b817a38a233fef676\": container with ID starting with ae6f548ae4112ea8d47998c67834c480598c4a605d79778b817a38a233fef676 not found: ID does not exist" containerID="ae6f548ae4112ea8d47998c67834c480598c4a605d79778b817a38a233fef676"
Jul 29 19:56:56.668736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.668674    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6f548ae4112ea8d47998c67834c480598c4a605d79778b817a38a233fef676"} err="failed to get container status \"ae6f548ae4112ea8d47998c67834c480598c4a605d79778b817a38a233fef676\": rpc error: code = NotFound desc = could not find container \"ae6f548ae4112ea8d47998c67834c480598c4a605d79778b817a38a233fef676\": container with ID starting with ae6f548ae4112ea8d47998c67834c480598c4a605d79778b817a38a233fef676 not found: ID does not exist"
Jul 29 19:56:56.668736 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.668691    2579 scope.go:117] "RemoveContainer" containerID="bb9f0ea1e7715a3039e0b5ffed1b134a7f3333c7993ce31e073c3a2371db4437"
Jul 29 19:56:56.677716 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.677684    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"]
Jul 29 19:56:56.678676 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.678659    2579 scope.go:117] "RemoveContainer" containerID="bb9f0ea1e7715a3039e0b5ffed1b134a7f3333c7993ce31e073c3a2371db4437"
Jul 29 19:56:56.679066 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:56:56.679038    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9f0ea1e7715a3039e0b5ffed1b134a7f3333c7993ce31e073c3a2371db4437\": container with ID starting with bb9f0ea1e7715a3039e0b5ffed1b134a7f3333c7993ce31e073c3a2371db4437 not found: ID does not exist" containerID="bb9f0ea1e7715a3039e0b5ffed1b134a7f3333c7993ce31e073c3a2371db4437"
Jul 29 19:56:56.679167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.679075    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9f0ea1e7715a3039e0b5ffed1b134a7f3333c7993ce31e073c3a2371db4437"} err="failed to get container status \"bb9f0ea1e7715a3039e0b5ffed1b134a7f3333c7993ce31e073c3a2371db4437\": rpc error: code = NotFound desc = could not find container \"bb9f0ea1e7715a3039e0b5ffed1b134a7f3333c7993ce31e073c3a2371db4437\": container with ID starting with bb9f0ea1e7715a3039e0b5ffed1b134a7f3333c7993ce31e073c3a2371db4437 not found: ID does not exist"
Jul 29 19:56:56.682258 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.682228    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-040fde-58688db7f4-r8mhg"]
Jul 29 19:56:56.695306 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.695218    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"]
Jul 29 19:56:56.701937 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.701905    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-040fde-5cbcbf7df4-ftlgn"]
Jul 29 19:56:56.713182 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.713147    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"]
Jul 29 19:56:56.718987 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:56.718955    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-040fde-fb9b8894d-8p2q7"]
Jul 29 19:56:58.197250 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:58.197186    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ee2996c-86dd-405f-a049-c909121ee879" path="/var/lib/kubelet/pods/0ee2996c-86dd-405f-a049-c909121ee879/volumes"
Jul 29 19:56:58.197647 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:58.197586    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54adcb4d-56f8-4912-a129-eff2763aa204" path="/var/lib/kubelet/pods/54adcb4d-56f8-4912-a129-eff2763aa204/volumes"
Jul 29 19:56:58.197899 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:56:58.197886    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbf28d22-bb2e-4151-8f9c-44eb0fc714b7" path="/var/lib/kubelet/pods/bbf28d22-bb2e-4151-8f9c-44eb0fc714b7/volumes"
Jul 29 19:57:21.798167 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:21.798129    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"]
Jul 29 19:57:21.798808 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:21.798467    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb" podUID="519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31" containerName="maas-api" containerID="cri-o://d57361a030485ba41a4aae149494a9167bb1384b63a21edd0bcac81985d87dee" gracePeriod=30
Jul 29 19:57:21.855453 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:21.855398    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"]
Jul 29 19:57:21.855835 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:21.855679    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj" podUID="195f9ec6-3a39-4ea4-864c-4a680c23e176" containerName="payload-processing" containerID="cri-o://aded5ad1c7e0da6b0af6c830a569da64ae16df28bf80b3f1e85aa26f6471d5a3" gracePeriod=30
Jul 29 19:57:21.857989 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:21.857956    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"]
Jul 29 19:57:21.858247 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:21.858213    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh" podUID="bbe4f538-6c3d-458b-b120-7ac00b311a39" containerName="payload-pre-processing" containerID="cri-o://91122e8d7ebda8ace26b82309ebe3ef38e98abce1e24900f90c0c803e87007b6" gracePeriod=30
Jul 29 19:57:21.858552 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:21.858473    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb" podUID="519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.86:8443/health\": dial tcp 10.134.0.86:8443: connect: connection refused"
Jul 29 19:57:22.124785 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.124747    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:57:22.165608 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.165571    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"
Jul 29 19:57:22.175631 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.175598    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"
Jul 29 19:57:22.208393 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.208356    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe4f538-6c3d-458b-b120-7ac00b311a39-plugins-config-volume\") pod \"bbe4f538-6c3d-458b-b120-7ac00b311a39\" (UID: \"bbe4f538-6c3d-458b-b120-7ac00b311a39\") "
Jul 29 19:57:22.208587 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.208434    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-maas-api-tls\") pod \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\" (UID: \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\") "
Jul 29 19:57:22.208587 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.208497    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx5xn\" (UniqueName: \"kubernetes.io/projected/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-kube-api-access-kx5xn\") pod \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\" (UID: \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\") "
Jul 29 19:57:22.208587 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.208554    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-openshift-service-ca\") pod \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\" (UID: \"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31\") "
Jul 29 19:57:22.208765 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.208597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpdvx\" (UniqueName: \"kubernetes.io/projected/bbe4f538-6c3d-458b-b120-7ac00b311a39-kube-api-access-rpdvx\") pod \"bbe4f538-6c3d-458b-b120-7ac00b311a39\" (UID: \"bbe4f538-6c3d-458b-b120-7ac00b311a39\") "
Jul 29 19:57:22.208765 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.208623    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195f9ec6-3a39-4ea4-864c-4a680c23e176-plugins-config-volume\") pod \"195f9ec6-3a39-4ea4-864c-4a680c23e176\" (UID: \"195f9ec6-3a39-4ea4-864c-4a680c23e176\") "
Jul 29 19:57:22.208765 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.208645    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k5lc\" (UniqueName: \"kubernetes.io/projected/195f9ec6-3a39-4ea4-864c-4a680c23e176-kube-api-access-6k5lc\") pod \"195f9ec6-3a39-4ea4-864c-4a680c23e176\" (UID: \"195f9ec6-3a39-4ea4-864c-4a680c23e176\") "
Jul 29 19:57:22.208765 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.208737    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe4f538-6c3d-458b-b120-7ac00b311a39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbe4f538-6c3d-458b-b120-7ac00b311a39" (UID: "bbe4f538-6c3d-458b-b120-7ac00b311a39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:57:22.209078 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.209057    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe4f538-6c3d-458b-b120-7ac00b311a39-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:57:22.209221 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.209180    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195f9ec6-3a39-4ea4-864c-4a680c23e176-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "195f9ec6-3a39-4ea4-864c-4a680c23e176" (UID: "195f9ec6-3a39-4ea4-864c-4a680c23e176"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:57:22.209818 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.209795    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31" (UID: "519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:57:22.211139 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.211110    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-kube-api-access-kx5xn" (OuterVolumeSpecName: "kube-api-access-kx5xn") pod "519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31" (UID: "519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31"). InnerVolumeSpecName "kube-api-access-kx5xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:57:22.211392 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.211365    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31" (UID: "519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:57:22.211589 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.211570    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe4f538-6c3d-458b-b120-7ac00b311a39-kube-api-access-rpdvx" (OuterVolumeSpecName: "kube-api-access-rpdvx") pod "bbe4f538-6c3d-458b-b120-7ac00b311a39" (UID: "bbe4f538-6c3d-458b-b120-7ac00b311a39"). InnerVolumeSpecName "kube-api-access-rpdvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:57:22.212063 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.212031    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195f9ec6-3a39-4ea4-864c-4a680c23e176-kube-api-access-6k5lc" (OuterVolumeSpecName: "kube-api-access-6k5lc") pod "195f9ec6-3a39-4ea4-864c-4a680c23e176" (UID: "195f9ec6-3a39-4ea4-864c-4a680c23e176"). InnerVolumeSpecName "kube-api-access-6k5lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:57:22.310596 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.310558    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:57:22.310596 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.310589    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx5xn\" (UniqueName: \"kubernetes.io/projected/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-kube-api-access-kx5xn\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:57:22.310596 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.310599    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:57:22.310850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.310609    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpdvx\" (UniqueName: \"kubernetes.io/projected/bbe4f538-6c3d-458b-b120-7ac00b311a39-kube-api-access-rpdvx\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:57:22.310850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.310620    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195f9ec6-3a39-4ea4-864c-4a680c23e176-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:57:22.310850 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.310632    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k5lc\" (UniqueName: \"kubernetes.io/projected/195f9ec6-3a39-4ea4-864c-4a680c23e176-kube-api-access-6k5lc\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:57:22.810707 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.810662    2579 generic.go:356] "Generic (PLEG): container finished" podID="bbe4f538-6c3d-458b-b120-7ac00b311a39" containerID="91122e8d7ebda8ace26b82309ebe3ef38e98abce1e24900f90c0c803e87007b6" exitCode=0
Jul 29 19:57:22.811186 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.810718    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"
Jul 29 19:57:22.811186 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.810708    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh" event={"ID":"bbe4f538-6c3d-458b-b120-7ac00b311a39","Type":"ContainerDied","Data":"91122e8d7ebda8ace26b82309ebe3ef38e98abce1e24900f90c0c803e87007b6"}
Jul 29 19:57:22.811186 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.810894    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh" event={"ID":"bbe4f538-6c3d-458b-b120-7ac00b311a39","Type":"ContainerDied","Data":"4a68842cac5eebc4821a1f26cd7b68c6fef391dcbbe16e7fa60b69ad61c2513d"}
Jul 29 19:57:22.811186 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.810951    2579 scope.go:117] "RemoveContainer" containerID="91122e8d7ebda8ace26b82309ebe3ef38e98abce1e24900f90c0c803e87007b6"
Jul 29 19:57:22.812842 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.812805    2579 generic.go:356] "Generic (PLEG): container finished" podID="519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31" containerID="d57361a030485ba41a4aae149494a9167bb1384b63a21edd0bcac81985d87dee" exitCode=0
Jul 29 19:57:22.812962 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.812892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb" event={"ID":"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31","Type":"ContainerDied","Data":"d57361a030485ba41a4aae149494a9167bb1384b63a21edd0bcac81985d87dee"}
Jul 29 19:57:22.812962 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.812947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb" event={"ID":"519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31","Type":"ContainerDied","Data":"49545120c6f3c15e865e4d8085ffadf1d227725325e59b29e41b5aacf6b194e7"}
Jul 29 19:57:22.813080 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.813002    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"
Jul 29 19:57:22.816371 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.814946    2579 generic.go:356] "Generic (PLEG): container finished" podID="195f9ec6-3a39-4ea4-864c-4a680c23e176" containerID="aded5ad1c7e0da6b0af6c830a569da64ae16df28bf80b3f1e85aa26f6471d5a3" exitCode=0
Jul 29 19:57:22.816371 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.815163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj" event={"ID":"195f9ec6-3a39-4ea4-864c-4a680c23e176","Type":"ContainerDied","Data":"aded5ad1c7e0da6b0af6c830a569da64ae16df28bf80b3f1e85aa26f6471d5a3"}
Jul 29 19:57:22.816371 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.815252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj" event={"ID":"195f9ec6-3a39-4ea4-864c-4a680c23e176","Type":"ContainerDied","Data":"4f32aec16cf94085b2c3e6a406ef133eb09cfeef2484303caecaedf0aace6612"}
Jul 29 19:57:22.816371 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.815360    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"
Jul 29 19:57:22.835481 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.834944    2579 scope.go:117] "RemoveContainer" containerID="91122e8d7ebda8ace26b82309ebe3ef38e98abce1e24900f90c0c803e87007b6"
Jul 29 19:57:22.835562 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:57:22.835501    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91122e8d7ebda8ace26b82309ebe3ef38e98abce1e24900f90c0c803e87007b6\": container with ID starting with 91122e8d7ebda8ace26b82309ebe3ef38e98abce1e24900f90c0c803e87007b6 not found: ID does not exist" containerID="91122e8d7ebda8ace26b82309ebe3ef38e98abce1e24900f90c0c803e87007b6"
Jul 29 19:57:22.835562 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.835537    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91122e8d7ebda8ace26b82309ebe3ef38e98abce1e24900f90c0c803e87007b6"} err="failed to get container status \"91122e8d7ebda8ace26b82309ebe3ef38e98abce1e24900f90c0c803e87007b6\": rpc error: code = NotFound desc = could not find container \"91122e8d7ebda8ace26b82309ebe3ef38e98abce1e24900f90c0c803e87007b6\": container with ID starting with 91122e8d7ebda8ace26b82309ebe3ef38e98abce1e24900f90c0c803e87007b6 not found: ID does not exist"
Jul 29 19:57:22.835669 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.835562    2579 scope.go:117] "RemoveContainer" containerID="d57361a030485ba41a4aae149494a9167bb1384b63a21edd0bcac81985d87dee"
Jul 29 19:57:22.846874 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.846853    2579 scope.go:117] "RemoveContainer" containerID="d57361a030485ba41a4aae149494a9167bb1384b63a21edd0bcac81985d87dee"
Jul 29 19:57:22.847293 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:57:22.847260    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57361a030485ba41a4aae149494a9167bb1384b63a21edd0bcac81985d87dee\": container with ID starting with d57361a030485ba41a4aae149494a9167bb1384b63a21edd0bcac81985d87dee not found: ID does not exist" containerID="d57361a030485ba41a4aae149494a9167bb1384b63a21edd0bcac81985d87dee"
Jul 29 19:57:22.847452 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.847301    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57361a030485ba41a4aae149494a9167bb1384b63a21edd0bcac81985d87dee"} err="failed to get container status \"d57361a030485ba41a4aae149494a9167bb1384b63a21edd0bcac81985d87dee\": rpc error: code = NotFound desc = could not find container \"d57361a030485ba41a4aae149494a9167bb1384b63a21edd0bcac81985d87dee\": container with ID starting with d57361a030485ba41a4aae149494a9167bb1384b63a21edd0bcac81985d87dee not found: ID does not exist"
Jul 29 19:57:22.847452 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.847327    2579 scope.go:117] "RemoveContainer" containerID="aded5ad1c7e0da6b0af6c830a569da64ae16df28bf80b3f1e85aa26f6471d5a3"
Jul 29 19:57:22.851525 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.851469    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"]
Jul 29 19:57:22.854548 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.854517    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d9f53-68955b9d8-p9tsh"]
Jul 29 19:57:22.858775 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.858751    2579 scope.go:117] "RemoveContainer" containerID="aded5ad1c7e0da6b0af6c830a569da64ae16df28bf80b3f1e85aa26f6471d5a3"
Jul 29 19:57:22.859111 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:57:22.859093    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aded5ad1c7e0da6b0af6c830a569da64ae16df28bf80b3f1e85aa26f6471d5a3\": container with ID starting with aded5ad1c7e0da6b0af6c830a569da64ae16df28bf80b3f1e85aa26f6471d5a3 not found: ID does not exist" containerID="aded5ad1c7e0da6b0af6c830a569da64ae16df28bf80b3f1e85aa26f6471d5a3"
Jul 29 19:57:22.859180 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.859118    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aded5ad1c7e0da6b0af6c830a569da64ae16df28bf80b3f1e85aa26f6471d5a3"} err="failed to get container status \"aded5ad1c7e0da6b0af6c830a569da64ae16df28bf80b3f1e85aa26f6471d5a3\": rpc error: code = NotFound desc = could not find container \"aded5ad1c7e0da6b0af6c830a569da64ae16df28bf80b3f1e85aa26f6471d5a3\": container with ID starting with aded5ad1c7e0da6b0af6c830a569da64ae16df28bf80b3f1e85aa26f6471d5a3 not found: ID does not exist"
Jul 29 19:57:22.866009 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.865974    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"]
Jul 29 19:57:22.871318 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.871276    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d9f53-8654c7dd5b-h44tb"]
Jul 29 19:57:22.882621 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.882572    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"]
Jul 29 19:57:22.886737 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:22.886699    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d9f53-8468456b54-lxpnj"]
Jul 29 19:57:24.197010 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:24.196936    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="195f9ec6-3a39-4ea4-864c-4a680c23e176" path="/var/lib/kubelet/pods/195f9ec6-3a39-4ea4-864c-4a680c23e176/volumes"
Jul 29 19:57:24.197387 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:24.197336    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31" path="/var/lib/kubelet/pods/519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31/volumes"
Jul 29 19:57:24.197647 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:57:24.197633    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbe4f538-6c3d-458b-b120-7ac00b311a39" path="/var/lib/kubelet/pods/bbe4f538-6c3d-458b-b120-7ac00b311a39/volumes"
Jul 29 19:58:21.487910 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.487871    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"]
Jul 29 19:58:21.488573 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.488112    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n" podUID="06e109d8-682c-4c9a-85c0-b4f177eee884" containerName="maas-api" containerID="cri-o://bc55dcaa970e9fc0551b673489cf47cf45799c82c4e5cbaaba1cfd10b2713f7c" gracePeriod=30
Jul 29 19:58:21.551293 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.551104    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"]
Jul 29 19:58:21.551512 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.551430    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47" podUID="689f313a-0533-4967-a237-08406df3b45b" containerName="payload-processing" containerID="cri-o://91ea4f6ff39a016af2b08ac930d3172c7cf40150e2c587afda9bfd07d1850bf5" gracePeriod=30
Jul 29 19:58:21.554563 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.554520    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"]
Jul 29 19:58:21.554987 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.554954    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2" podUID="7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3" containerName="payload-pre-processing" containerID="cri-o://a466e497ab2500a554d7f0da808775937d1b85ab2608764e1ee2887fde80938e" gracePeriod=30
Jul 29 19:58:21.818506 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.818479    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:58:21.855222 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.855168    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"
Jul 29 19:58:21.870081 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.870052    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"
Jul 29 19:58:21.884515 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.884486    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/06e109d8-682c-4c9a-85c0-b4f177eee884-kube-api-access-m5pgv\") pod \"06e109d8-682c-4c9a-85c0-b4f177eee884\" (UID: \"06e109d8-682c-4c9a-85c0-b4f177eee884\") "
Jul 29 19:58:21.884673 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.884544    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e109d8-682c-4c9a-85c0-b4f177eee884-openshift-service-ca\") pod \"06e109d8-682c-4c9a-85c0-b4f177eee884\" (UID: \"06e109d8-682c-4c9a-85c0-b4f177eee884\") "
Jul 29 19:58:21.884673 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.884630    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e109d8-682c-4c9a-85c0-b4f177eee884-maas-api-tls\") pod \"06e109d8-682c-4c9a-85c0-b4f177eee884\" (UID: \"06e109d8-682c-4c9a-85c0-b4f177eee884\") "
Jul 29 19:58:21.884994 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.884964    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e109d8-682c-4c9a-85c0-b4f177eee884-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06e109d8-682c-4c9a-85c0-b4f177eee884" (UID: "06e109d8-682c-4c9a-85c0-b4f177eee884"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:21.886880 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.886851    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e109d8-682c-4c9a-85c0-b4f177eee884-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06e109d8-682c-4c9a-85c0-b4f177eee884" (UID: "06e109d8-682c-4c9a-85c0-b4f177eee884"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:58:21.886991 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.886929    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e109d8-682c-4c9a-85c0-b4f177eee884-kube-api-access-m5pgv" (OuterVolumeSpecName: "kube-api-access-m5pgv") pod "06e109d8-682c-4c9a-85c0-b4f177eee884" (UID: "06e109d8-682c-4c9a-85c0-b4f177eee884"). InnerVolumeSpecName "kube-api-access-m5pgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:21.985404 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.985310    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtq8n\" (UniqueName: \"kubernetes.io/projected/689f313a-0533-4967-a237-08406df3b45b-kube-api-access-rtq8n\") pod \"689f313a-0533-4967-a237-08406df3b45b\" (UID: \"689f313a-0533-4967-a237-08406df3b45b\") "
Jul 29 19:58:21.985560 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.985455    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkl7c\" (UniqueName: \"kubernetes.io/projected/7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3-kube-api-access-wkl7c\") pod \"7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3\" (UID: \"7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3\") "
Jul 29 19:58:21.985560 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.985510    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3-plugins-config-volume\") pod \"7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3\" (UID: \"7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3\") "
Jul 29 19:58:21.985671 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.985608    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689f313a-0533-4967-a237-08406df3b45b-plugins-config-volume\") pod \"689f313a-0533-4967-a237-08406df3b45b\" (UID: \"689f313a-0533-4967-a237-08406df3b45b\") "
Jul 29 19:58:21.985905 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.985886    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06e109d8-682c-4c9a-85c0-b4f177eee884-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:21.985989 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.985908    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/06e109d8-682c-4c9a-85c0-b4f177eee884-kube-api-access-m5pgv\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:21.985989 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.985920    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e109d8-682c-4c9a-85c0-b4f177eee884-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:21.985989 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.985936    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3" (UID: "7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:21.985989 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.985956    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689f313a-0533-4967-a237-08406df3b45b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "689f313a-0533-4967-a237-08406df3b45b" (UID: "689f313a-0533-4967-a237-08406df3b45b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:21.987718 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.987696    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3-kube-api-access-wkl7c" (OuterVolumeSpecName: "kube-api-access-wkl7c") pod "7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3" (UID: "7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3"). InnerVolumeSpecName "kube-api-access-wkl7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:21.987718 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:21.987709    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689f313a-0533-4967-a237-08406df3b45b-kube-api-access-rtq8n" (OuterVolumeSpecName: "kube-api-access-rtq8n") pod "689f313a-0533-4967-a237-08406df3b45b" (UID: "689f313a-0533-4967-a237-08406df3b45b"). InnerVolumeSpecName "kube-api-access-rtq8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:22.086984 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.086895    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:22.086984 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.086924    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689f313a-0533-4967-a237-08406df3b45b-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:22.086984 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.086937    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtq8n\" (UniqueName: \"kubernetes.io/projected/689f313a-0533-4967-a237-08406df3b45b-kube-api-access-rtq8n\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:22.086984 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.086945    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkl7c\" (UniqueName: \"kubernetes.io/projected/7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3-kube-api-access-wkl7c\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:22.139562 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.139525    2579 generic.go:356] "Generic (PLEG): container finished" podID="06e109d8-682c-4c9a-85c0-b4f177eee884" containerID="bc55dcaa970e9fc0551b673489cf47cf45799c82c4e5cbaaba1cfd10b2713f7c" exitCode=0
Jul 29 19:58:22.139759 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.139574    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"
Jul 29 19:58:22.139759 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.139615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n" event={"ID":"06e109d8-682c-4c9a-85c0-b4f177eee884","Type":"ContainerDied","Data":"bc55dcaa970e9fc0551b673489cf47cf45799c82c4e5cbaaba1cfd10b2713f7c"}
Jul 29 19:58:22.139759 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.139673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n" event={"ID":"06e109d8-682c-4c9a-85c0-b4f177eee884","Type":"ContainerDied","Data":"ac93082999e9367fbf263793a2e159071f472681b6600fbba85abd935fa82dff"}
Jul 29 19:58:22.139759 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.139700    2579 scope.go:117] "RemoveContainer" containerID="bc55dcaa970e9fc0551b673489cf47cf45799c82c4e5cbaaba1cfd10b2713f7c"
Jul 29 19:58:22.141108 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.141078    2579 generic.go:356] "Generic (PLEG): container finished" podID="689f313a-0533-4967-a237-08406df3b45b" containerID="91ea4f6ff39a016af2b08ac930d3172c7cf40150e2c587afda9bfd07d1850bf5" exitCode=0
Jul 29 19:58:22.141242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.141133    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"
Jul 29 19:58:22.141242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.141161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47" event={"ID":"689f313a-0533-4967-a237-08406df3b45b","Type":"ContainerDied","Data":"91ea4f6ff39a016af2b08ac930d3172c7cf40150e2c587afda9bfd07d1850bf5"}
Jul 29 19:58:22.141242 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.141234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47" event={"ID":"689f313a-0533-4967-a237-08406df3b45b","Type":"ContainerDied","Data":"35c67263920a395dbc7c200d97251c527c8b562b675fbfb267a944cf196841b6"}
Jul 29 19:58:22.142968 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.142942    2579 generic.go:356] "Generic (PLEG): container finished" podID="7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3" containerID="a466e497ab2500a554d7f0da808775937d1b85ab2608764e1ee2887fde80938e" exitCode=0
Jul 29 19:58:22.143102 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.142985    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"
Jul 29 19:58:22.143102 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.143038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2" event={"ID":"7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3","Type":"ContainerDied","Data":"a466e497ab2500a554d7f0da808775937d1b85ab2608764e1ee2887fde80938e"}
Jul 29 19:58:22.143102 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.143080    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2" event={"ID":"7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3","Type":"ContainerDied","Data":"5574c41c598f868e210305330d5b820670a7450f29dbc8b4910d3be51f88a319"}
Jul 29 19:58:22.149812 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.149787    2579 scope.go:117] "RemoveContainer" containerID="bc55dcaa970e9fc0551b673489cf47cf45799c82c4e5cbaaba1cfd10b2713f7c"
Jul 29 19:58:22.150307 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:58:22.150262    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc55dcaa970e9fc0551b673489cf47cf45799c82c4e5cbaaba1cfd10b2713f7c\": container with ID starting with bc55dcaa970e9fc0551b673489cf47cf45799c82c4e5cbaaba1cfd10b2713f7c not found: ID does not exist" containerID="bc55dcaa970e9fc0551b673489cf47cf45799c82c4e5cbaaba1cfd10b2713f7c"
Jul 29 19:58:22.150427 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.150308    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc55dcaa970e9fc0551b673489cf47cf45799c82c4e5cbaaba1cfd10b2713f7c"} err="failed to get container status \"bc55dcaa970e9fc0551b673489cf47cf45799c82c4e5cbaaba1cfd10b2713f7c\": rpc error: code = NotFound desc = could not find container \"bc55dcaa970e9fc0551b673489cf47cf45799c82c4e5cbaaba1cfd10b2713f7c\": container with ID starting with bc55dcaa970e9fc0551b673489cf47cf45799c82c4e5cbaaba1cfd10b2713f7c not found: ID does not exist"
Jul 29 19:58:22.150427 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.150341    2579 scope.go:117] "RemoveContainer" containerID="91ea4f6ff39a016af2b08ac930d3172c7cf40150e2c587afda9bfd07d1850bf5"
Jul 29 19:58:22.160328 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.160307    2579 scope.go:117] "RemoveContainer" containerID="91ea4f6ff39a016af2b08ac930d3172c7cf40150e2c587afda9bfd07d1850bf5"
Jul 29 19:58:22.160686 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:58:22.160665    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ea4f6ff39a016af2b08ac930d3172c7cf40150e2c587afda9bfd07d1850bf5\": container with ID starting with 91ea4f6ff39a016af2b08ac930d3172c7cf40150e2c587afda9bfd07d1850bf5 not found: ID does not exist" containerID="91ea4f6ff39a016af2b08ac930d3172c7cf40150e2c587afda9bfd07d1850bf5"
Jul 29 19:58:22.160741 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.160695    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ea4f6ff39a016af2b08ac930d3172c7cf40150e2c587afda9bfd07d1850bf5"} err="failed to get container status \"91ea4f6ff39a016af2b08ac930d3172c7cf40150e2c587afda9bfd07d1850bf5\": rpc error: code = NotFound desc = could not find container \"91ea4f6ff39a016af2b08ac930d3172c7cf40150e2c587afda9bfd07d1850bf5\": container with ID starting with 91ea4f6ff39a016af2b08ac930d3172c7cf40150e2c587afda9bfd07d1850bf5 not found: ID does not exist"
Jul 29 19:58:22.160741 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.160715    2579 scope.go:117] "RemoveContainer" containerID="a466e497ab2500a554d7f0da808775937d1b85ab2608764e1ee2887fde80938e"
Jul 29 19:58:22.169939 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.169904    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"]
Jul 29 19:58:22.170217 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.170172    2579 scope.go:117] "RemoveContainer" containerID="a466e497ab2500a554d7f0da808775937d1b85ab2608764e1ee2887fde80938e"
Jul 29 19:58:22.170551 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:58:22.170524    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a466e497ab2500a554d7f0da808775937d1b85ab2608764e1ee2887fde80938e\": container with ID starting with a466e497ab2500a554d7f0da808775937d1b85ab2608764e1ee2887fde80938e not found: ID does not exist" containerID="a466e497ab2500a554d7f0da808775937d1b85ab2608764e1ee2887fde80938e"
Jul 29 19:58:22.170589 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.170564    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a466e497ab2500a554d7f0da808775937d1b85ab2608764e1ee2887fde80938e"} err="failed to get container status \"a466e497ab2500a554d7f0da808775937d1b85ab2608764e1ee2887fde80938e\": rpc error: code = NotFound desc = could not find container \"a466e497ab2500a554d7f0da808775937d1b85ab2608764e1ee2887fde80938e\": container with ID starting with a466e497ab2500a554d7f0da808775937d1b85ab2608764e1ee2887fde80938e not found: ID does not exist"
Jul 29 19:58:22.174065 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.174038    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f36365-59c7845f76-hnx7n"]
Jul 29 19:58:22.185925 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.185893    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"]
Jul 29 19:58:22.190258 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.190229    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f36365-6dddcbb4d6-k2k47"]
Jul 29 19:58:22.218696 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.218659    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06e109d8-682c-4c9a-85c0-b4f177eee884" path="/var/lib/kubelet/pods/06e109d8-682c-4c9a-85c0-b4f177eee884/volumes"
Jul 29 19:58:22.219208 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.219175    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="689f313a-0533-4967-a237-08406df3b45b" path="/var/lib/kubelet/pods/689f313a-0533-4967-a237-08406df3b45b/volumes"
Jul 29 19:58:22.219588 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.219573    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"]
Jul 29 19:58:22.222422 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:22.222399    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f36365-6756d8c85b-rffx2"]
Jul 29 19:58:24.196629 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:24.196598    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3" path="/var/lib/kubelet/pods/7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3/volumes"
Jul 29 19:58:39.039437 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.039389    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"]
Jul 29 19:58:39.039861 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.039659    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw" podUID="1e264efc-fae3-459d-9a39-cf4c9f8fdae8" containerName="maas-api" containerID="cri-o://e76c9bff4fb48276eace673d672285c37b09e78b65b59223f9a4276f02dcc4f3" gracePeriod=30
Jul 29 19:58:39.088816 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.088786    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"]
Jul 29 19:58:39.089048 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.089007    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt" podUID="5edd5028-1cb7-4213-aa68-205703a7e339" containerName="payload-processing" containerID="cri-o://8cef3701e28baa5e23e9002672c5238306c179808dac679eaf07b46adf06e351" gracePeriod=30
Jul 29 19:58:39.096525 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.096497    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"]
Jul 29 19:58:39.096906 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.096723    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg" podUID="e029d7af-1acb-4676-af97-c16fc92d8c20" containerName="payload-pre-processing" containerID="cri-o://e9e63ebb1e66e93dba01506337deadd355ad2290f17bad635e301153ff7be9c3" gracePeriod=30
Jul 29 19:58:39.236977 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.236930    2579 generic.go:356] "Generic (PLEG): container finished" podID="e029d7af-1acb-4676-af97-c16fc92d8c20" containerID="e9e63ebb1e66e93dba01506337deadd355ad2290f17bad635e301153ff7be9c3" exitCode=0
Jul 29 19:58:39.237168 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.237013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg" event={"ID":"e029d7af-1acb-4676-af97-c16fc92d8c20","Type":"ContainerDied","Data":"e9e63ebb1e66e93dba01506337deadd355ad2290f17bad635e301153ff7be9c3"}
Jul 29 19:58:39.238900 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.238872    2579 generic.go:356] "Generic (PLEG): container finished" podID="1e264efc-fae3-459d-9a39-cf4c9f8fdae8" containerID="e76c9bff4fb48276eace673d672285c37b09e78b65b59223f9a4276f02dcc4f3" exitCode=0
Jul 29 19:58:39.239063 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.238947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw" event={"ID":"1e264efc-fae3-459d-9a39-cf4c9f8fdae8","Type":"ContainerDied","Data":"e76c9bff4fb48276eace673d672285c37b09e78b65b59223f9a4276f02dcc4f3"}
Jul 29 19:58:39.240629 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.240596    2579 generic.go:356] "Generic (PLEG): container finished" podID="5edd5028-1cb7-4213-aa68-205703a7e339" containerID="8cef3701e28baa5e23e9002672c5238306c179808dac679eaf07b46adf06e351" exitCode=0
Jul 29 19:58:39.240773 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.240754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt" event={"ID":"5edd5028-1cb7-4213-aa68-205703a7e339","Type":"ContainerDied","Data":"8cef3701e28baa5e23e9002672c5238306c179808dac679eaf07b46adf06e351"}
Jul 29 19:58:39.379869 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.379846    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"
Jul 29 19:58:39.383083 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.383064    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:58:39.408656 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.408636    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"
Jul 29 19:58:39.552513 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.552485    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-maas-api-tls\") pod \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\" (UID: \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\") "
Jul 29 19:58:39.552710 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.552549    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-openshift-service-ca\") pod \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\" (UID: \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\") "
Jul 29 19:58:39.552710 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.552571    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edd5028-1cb7-4213-aa68-205703a7e339-plugins-config-volume\") pod \"5edd5028-1cb7-4213-aa68-205703a7e339\" (UID: \"5edd5028-1cb7-4213-aa68-205703a7e339\") "
Jul 29 19:58:39.552710 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.552597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v47hv\" (UniqueName: \"kubernetes.io/projected/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-kube-api-access-v47hv\") pod \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\" (UID: \"1e264efc-fae3-459d-9a39-cf4c9f8fdae8\") "
Jul 29 19:58:39.552710 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.552612    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shrrx\" (UniqueName: \"kubernetes.io/projected/5edd5028-1cb7-4213-aa68-205703a7e339-kube-api-access-shrrx\") pod \"5edd5028-1cb7-4213-aa68-205703a7e339\" (UID: \"5edd5028-1cb7-4213-aa68-205703a7e339\") "
Jul 29 19:58:39.552710 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.552633    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e029d7af-1acb-4676-af97-c16fc92d8c20-plugins-config-volume\") pod \"e029d7af-1acb-4676-af97-c16fc92d8c20\" (UID: \"e029d7af-1acb-4676-af97-c16fc92d8c20\") "
Jul 29 19:58:39.552710 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.552667    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cfdx\" (UniqueName: \"kubernetes.io/projected/e029d7af-1acb-4676-af97-c16fc92d8c20-kube-api-access-8cfdx\") pod \"e029d7af-1acb-4676-af97-c16fc92d8c20\" (UID: \"e029d7af-1acb-4676-af97-c16fc92d8c20\") "
Jul 29 19:58:39.553050 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.553017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e264efc-fae3-459d-9a39-cf4c9f8fdae8" (UID: "1e264efc-fae3-459d-9a39-cf4c9f8fdae8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:39.553108 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.553042    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edd5028-1cb7-4213-aa68-205703a7e339-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5edd5028-1cb7-4213-aa68-205703a7e339" (UID: "5edd5028-1cb7-4213-aa68-205703a7e339"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:39.553108 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.553046    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e029d7af-1acb-4676-af97-c16fc92d8c20-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e029d7af-1acb-4676-af97-c16fc92d8c20" (UID: "e029d7af-1acb-4676-af97-c16fc92d8c20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:39.554782 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.554758    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e264efc-fae3-459d-9a39-cf4c9f8fdae8" (UID: "1e264efc-fae3-459d-9a39-cf4c9f8fdae8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:58:39.554980 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.554965    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e029d7af-1acb-4676-af97-c16fc92d8c20-kube-api-access-8cfdx" (OuterVolumeSpecName: "kube-api-access-8cfdx") pod "e029d7af-1acb-4676-af97-c16fc92d8c20" (UID: "e029d7af-1acb-4676-af97-c16fc92d8c20"). InnerVolumeSpecName "kube-api-access-8cfdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:39.555323 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.555294    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edd5028-1cb7-4213-aa68-205703a7e339-kube-api-access-shrrx" (OuterVolumeSpecName: "kube-api-access-shrrx") pod "5edd5028-1cb7-4213-aa68-205703a7e339" (UID: "5edd5028-1cb7-4213-aa68-205703a7e339"). InnerVolumeSpecName "kube-api-access-shrrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:39.555423 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.555326    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-kube-api-access-v47hv" (OuterVolumeSpecName: "kube-api-access-v47hv") pod "1e264efc-fae3-459d-9a39-cf4c9f8fdae8" (UID: "1e264efc-fae3-459d-9a39-cf4c9f8fdae8"). InnerVolumeSpecName "kube-api-access-v47hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:39.653988 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.653952    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:39.653988 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.653986    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:39.654229 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.654009    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edd5028-1cb7-4213-aa68-205703a7e339-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:39.654229 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.654020    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v47hv\" (UniqueName: \"kubernetes.io/projected/1e264efc-fae3-459d-9a39-cf4c9f8fdae8-kube-api-access-v47hv\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:39.654229 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.654030    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shrrx\" (UniqueName: \"kubernetes.io/projected/5edd5028-1cb7-4213-aa68-205703a7e339-kube-api-access-shrrx\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:39.654229 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.654039    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e029d7af-1acb-4676-af97-c16fc92d8c20-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:39.654229 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:39.654047    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cfdx\" (UniqueName: \"kubernetes.io/projected/e029d7af-1acb-4676-af97-c16fc92d8c20-kube-api-access-8cfdx\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:40.253535 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.253499    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"
Jul 29 19:58:40.254005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.253526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg" event={"ID":"e029d7af-1acb-4676-af97-c16fc92d8c20","Type":"ContainerDied","Data":"5778cdd7b68c4ab43ce1e29bdf8931e9b9d8e03ffb9580d85fe1de42c1d50024"}
Jul 29 19:58:40.254005 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.253630    2579 scope.go:117] "RemoveContainer" containerID="e9e63ebb1e66e93dba01506337deadd355ad2290f17bad635e301153ff7be9c3"
Jul 29 19:58:40.255174 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.255154    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"
Jul 29 19:58:40.255174 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.255162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw" event={"ID":"1e264efc-fae3-459d-9a39-cf4c9f8fdae8","Type":"ContainerDied","Data":"d3e0a293510c29853caf4d15903d702750e66d6824667cfeb4230f4c4ce310df"}
Jul 29 19:58:40.257294 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.257249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt" event={"ID":"5edd5028-1cb7-4213-aa68-205703a7e339","Type":"ContainerDied","Data":"f1ec04d8423290cc000a506a8b2fe9391324ebdbeb2940e452220ee32e05998d"}
Jul 29 19:58:40.257424 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.257406    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"
Jul 29 19:58:40.264046 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.264027    2579 scope.go:117] "RemoveContainer" containerID="e76c9bff4fb48276eace673d672285c37b09e78b65b59223f9a4276f02dcc4f3"
Jul 29 19:58:40.272073 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.272050    2579 scope.go:117] "RemoveContainer" containerID="8cef3701e28baa5e23e9002672c5238306c179808dac679eaf07b46adf06e351"
Jul 29 19:58:40.277998 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.277974    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"]
Jul 29 19:58:40.281517 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.281498    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-8e241a-664b5975d-cpfwt"]
Jul 29 19:58:40.293551 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.293524    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"]
Jul 29 19:58:40.298320 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.298286    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e241a-79f65cbc-wxhkw"]
Jul 29 19:58:40.308473 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.308399    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"]
Jul 29 19:58:40.312577 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:40.312549    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-8e241a-84cdfd5bdd-v2pxg"]
Jul 29 19:58:42.196213 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:42.196158    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e264efc-fae3-459d-9a39-cf4c9f8fdae8" path="/var/lib/kubelet/pods/1e264efc-fae3-459d-9a39-cf4c9f8fdae8/volumes"
Jul 29 19:58:42.196578 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:42.196540    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5edd5028-1cb7-4213-aa68-205703a7e339" path="/var/lib/kubelet/pods/5edd5028-1cb7-4213-aa68-205703a7e339/volumes"
Jul 29 19:58:42.196853 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:42.196840    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e029d7af-1acb-4676-af97-c16fc92d8c20" path="/var/lib/kubelet/pods/e029d7af-1acb-4676-af97-c16fc92d8c20/volumes"
Jul 29 19:58:56.543868 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.543768    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"]
Jul 29 19:58:56.544526 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.544055    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn" podUID="6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8" containerName="maas-api" containerID="cri-o://bc2d86fd33546e514b8e12af625fc653bfa85a0c0b8a7d6653add1beaef3a9bb" gracePeriod=30
Jul 29 19:58:56.592303 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.592263    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"]
Jul 29 19:58:56.592531 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.592484    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb" podUID="de79cbb9-8dfb-4a36-a5dc-c592668c55bb" containerName="payload-processing" containerID="cri-o://46f96f319974d530f38fc2869f438da6c55bed698beff08b3ad43920526bfba3" gracePeriod=30
Jul 29 19:58:56.599729 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.599668    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"]
Jul 29 19:58:56.600030 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.599998    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr" podUID="3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341" containerName="payload-pre-processing" containerID="cri-o://dd854b18d00e579aca1250be87abd091db9b298ad8b7fdaf0b2b5fe3920e518a" gracePeriod=30
Jul 29 19:58:56.865349 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.865316    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:58:56.899527 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.899500    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"
Jul 29 19:58:56.900534 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.900511    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxqfv\" (UniqueName: \"kubernetes.io/projected/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-kube-api-access-kxqfv\") pod \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\" (UID: \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\") "
Jul 29 19:58:56.900677 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.900560    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-maas-api-tls\") pod \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\" (UID: \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\") "
Jul 29 19:58:56.900677 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.900626    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-openshift-service-ca\") pod \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\" (UID: \"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8\") "
Jul 29 19:58:56.901138 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.901106    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8" (UID: "6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:56.902920 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.902894    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8" (UID: "6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:58:56.903248 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.903224    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-kube-api-access-kxqfv" (OuterVolumeSpecName: "kube-api-access-kxqfv") pod "6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8" (UID: "6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8"). InnerVolumeSpecName "kube-api-access-kxqfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:56.911540 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:56.911513    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"
Jul 29 19:58:57.001138 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.001099    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rq78j\" (UniqueName: \"kubernetes.io/projected/3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341-kube-api-access-rq78j\") pod \"3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341\" (UID: \"3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341\") "
Jul 29 19:58:57.001138 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.001147    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-685pl\" (UniqueName: \"kubernetes.io/projected/de79cbb9-8dfb-4a36-a5dc-c592668c55bb-kube-api-access-685pl\") pod \"de79cbb9-8dfb-4a36-a5dc-c592668c55bb\" (UID: \"de79cbb9-8dfb-4a36-a5dc-c592668c55bb\") "
Jul 29 19:58:57.001444 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.001296    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341-plugins-config-volume\") pod \"3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341\" (UID: \"3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341\") "
Jul 29 19:58:57.001444 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.001381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de79cbb9-8dfb-4a36-a5dc-c592668c55bb-plugins-config-volume\") pod \"de79cbb9-8dfb-4a36-a5dc-c592668c55bb\" (UID: \"de79cbb9-8dfb-4a36-a5dc-c592668c55bb\") "
Jul 29 19:58:57.001684 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.001665    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxqfv\" (UniqueName: \"kubernetes.io/projected/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-kube-api-access-kxqfv\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:57.001749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.001698    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-maas-api-tls\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:57.001749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.001714    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8-openshift-service-ca\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:57.001749 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.001661    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341" (UID: "3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:57.001854 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.001773    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de79cbb9-8dfb-4a36-a5dc-c592668c55bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de79cbb9-8dfb-4a36-a5dc-c592668c55bb" (UID: "de79cbb9-8dfb-4a36-a5dc-c592668c55bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:57.003449 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.003427    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341-kube-api-access-rq78j" (OuterVolumeSpecName: "kube-api-access-rq78j") pod "3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341" (UID: "3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341"). InnerVolumeSpecName "kube-api-access-rq78j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:57.003502 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.003477    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de79cbb9-8dfb-4a36-a5dc-c592668c55bb-kube-api-access-685pl" (OuterVolumeSpecName: "kube-api-access-685pl") pod "de79cbb9-8dfb-4a36-a5dc-c592668c55bb" (UID: "de79cbb9-8dfb-4a36-a5dc-c592668c55bb"). InnerVolumeSpecName "kube-api-access-685pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:57.103068 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.102970    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de79cbb9-8dfb-4a36-a5dc-c592668c55bb-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:57.103068 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.103002    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq78j\" (UniqueName: \"kubernetes.io/projected/3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341-kube-api-access-rq78j\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:57.103068 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.103013    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-685pl\" (UniqueName: \"kubernetes.io/projected/de79cbb9-8dfb-4a36-a5dc-c592668c55bb-kube-api-access-685pl\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:57.103068 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.103022    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341-plugins-config-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\""
Jul 29 19:58:57.346471 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.346435    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341" containerID="dd854b18d00e579aca1250be87abd091db9b298ad8b7fdaf0b2b5fe3920e518a" exitCode=0
Jul 29 19:58:57.346714 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.346487    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"
Jul 29 19:58:57.346714 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.346513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr" event={"ID":"3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341","Type":"ContainerDied","Data":"dd854b18d00e579aca1250be87abd091db9b298ad8b7fdaf0b2b5fe3920e518a"}
Jul 29 19:58:57.346714 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.346570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr" event={"ID":"3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341","Type":"ContainerDied","Data":"fb7cc5b202f25606313897aa330720e79145211b72a513257a36f23f63f13de2"}
Jul 29 19:58:57.346714 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.346596    2579 scope.go:117] "RemoveContainer" containerID="dd854b18d00e579aca1250be87abd091db9b298ad8b7fdaf0b2b5fe3920e518a"
Jul 29 19:58:57.347934 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.347910    2579 generic.go:356] "Generic (PLEG): container finished" podID="de79cbb9-8dfb-4a36-a5dc-c592668c55bb" containerID="46f96f319974d530f38fc2869f438da6c55bed698beff08b3ad43920526bfba3" exitCode=0
Jul 29 19:58:57.348070 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.347940    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb" event={"ID":"de79cbb9-8dfb-4a36-a5dc-c592668c55bb","Type":"ContainerDied","Data":"46f96f319974d530f38fc2869f438da6c55bed698beff08b3ad43920526bfba3"}
Jul 29 19:58:57.348070 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.347970    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"
Jul 29 19:58:57.348070 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.347984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb" event={"ID":"de79cbb9-8dfb-4a36-a5dc-c592668c55bb","Type":"ContainerDied","Data":"4b44382a971c3e62a3ff588d7439141966bc7840dcd5fb1ab3a3e75de174710d"}
Jul 29 19:58:57.349501 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.349475    2579 generic.go:356] "Generic (PLEG): container finished" podID="6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8" containerID="bc2d86fd33546e514b8e12af625fc653bfa85a0c0b8a7d6653add1beaef3a9bb" exitCode=0
Jul 29 19:58:57.349606 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.349521    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"
Jul 29 19:58:57.349606 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.349525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn" event={"ID":"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8","Type":"ContainerDied","Data":"bc2d86fd33546e514b8e12af625fc653bfa85a0c0b8a7d6653add1beaef3a9bb"}
Jul 29 19:58:57.349606 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.349579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn" event={"ID":"6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8","Type":"ContainerDied","Data":"464eb33bde4ae7eda8e0db56a623b37fb525d3557f08fe309945755ad6524a32"}
Jul 29 19:58:57.356547 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.356527    2579 scope.go:117] "RemoveContainer" containerID="dd854b18d00e579aca1250be87abd091db9b298ad8b7fdaf0b2b5fe3920e518a"
Jul 29 19:58:57.356848 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:58:57.356825    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd854b18d00e579aca1250be87abd091db9b298ad8b7fdaf0b2b5fe3920e518a\": container with ID starting with dd854b18d00e579aca1250be87abd091db9b298ad8b7fdaf0b2b5fe3920e518a not found: ID does not exist" containerID="dd854b18d00e579aca1250be87abd091db9b298ad8b7fdaf0b2b5fe3920e518a"
Jul 29 19:58:57.356915 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.356859    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd854b18d00e579aca1250be87abd091db9b298ad8b7fdaf0b2b5fe3920e518a"} err="failed to get container status \"dd854b18d00e579aca1250be87abd091db9b298ad8b7fdaf0b2b5fe3920e518a\": rpc error: code = NotFound desc = could not find container \"dd854b18d00e579aca1250be87abd091db9b298ad8b7fdaf0b2b5fe3920e518a\": container with ID starting with dd854b18d00e579aca1250be87abd091db9b298ad8b7fdaf0b2b5fe3920e518a not found: ID does not exist"
Jul 29 19:58:57.356915 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.356881    2579 scope.go:117] "RemoveContainer" containerID="46f96f319974d530f38fc2869f438da6c55bed698beff08b3ad43920526bfba3"
Jul 29 19:58:57.366637 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.366621    2579 scope.go:117] "RemoveContainer" containerID="46f96f319974d530f38fc2869f438da6c55bed698beff08b3ad43920526bfba3"
Jul 29 19:58:57.366882 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:58:57.366863    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f96f319974d530f38fc2869f438da6c55bed698beff08b3ad43920526bfba3\": container with ID starting with 46f96f319974d530f38fc2869f438da6c55bed698beff08b3ad43920526bfba3 not found: ID does not exist" containerID="46f96f319974d530f38fc2869f438da6c55bed698beff08b3ad43920526bfba3"
Jul 29 19:58:57.366937 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.366890    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f96f319974d530f38fc2869f438da6c55bed698beff08b3ad43920526bfba3"} err="failed to get container status \"46f96f319974d530f38fc2869f438da6c55bed698beff08b3ad43920526bfba3\": rpc error: code = NotFound desc = could not find container \"46f96f319974d530f38fc2869f438da6c55bed698beff08b3ad43920526bfba3\": container with ID starting with 46f96f319974d530f38fc2869f438da6c55bed698beff08b3ad43920526bfba3 not found: ID does not exist"
Jul 29 19:58:57.366937 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.366906    2579 scope.go:117] "RemoveContainer" containerID="bc2d86fd33546e514b8e12af625fc653bfa85a0c0b8a7d6653add1beaef3a9bb"
Jul 29 19:58:57.375053 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.375033    2579 scope.go:117] "RemoveContainer" containerID="bc2d86fd33546e514b8e12af625fc653bfa85a0c0b8a7d6653add1beaef3a9bb"
Jul 29 19:58:57.375472 ip-10-0-135-248 kubenswrapper[2579]: E0729 19:58:57.375436    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2d86fd33546e514b8e12af625fc653bfa85a0c0b8a7d6653add1beaef3a9bb\": container with ID starting with bc2d86fd33546e514b8e12af625fc653bfa85a0c0b8a7d6653add1beaef3a9bb not found: ID does not exist" containerID="bc2d86fd33546e514b8e12af625fc653bfa85a0c0b8a7d6653add1beaef3a9bb"
Jul 29 19:58:57.375545 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.375483    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2d86fd33546e514b8e12af625fc653bfa85a0c0b8a7d6653add1beaef3a9bb"} err="failed to get container status \"bc2d86fd33546e514b8e12af625fc653bfa85a0c0b8a7d6653add1beaef3a9bb\": rpc error: code = NotFound desc = could not find container \"bc2d86fd33546e514b8e12af625fc653bfa85a0c0b8a7d6653add1beaef3a9bb\": container with ID starting with bc2d86fd33546e514b8e12af625fc653bfa85a0c0b8a7d6653add1beaef3a9bb not found: ID does not exist"
Jul 29 19:58:57.376439 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.376420    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"]
Jul 29 19:58:57.382016 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.381993    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-808386-6dbbdc7698-dklmb"]
Jul 29 19:58:57.393623 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.393553    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"]
Jul 29 19:58:57.395306 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.395287    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-808386-6c59b7dbc6-nq7jr"]
Jul 29 19:58:57.408479 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.408449    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"]
Jul 29 19:58:57.412365 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:57.412339    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-808386-6dcd9747bd-6x8hn"]
Jul 29 19:58:58.195956 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:58.195924    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341" path="/var/lib/kubelet/pods/3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341/volumes"
Jul 29 19:58:58.196386 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:58.196352    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8" path="/var/lib/kubelet/pods/6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8/volumes"
Jul 29 19:58:58.196687 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:58:58.196673    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de79cbb9-8dfb-4a36-a5dc-c592668c55bb" path="/var/lib/kubelet/pods/de79cbb9-8dfb-4a36-a5dc-c592668c55bb/volumes"
Jul 29 19:59:09.834116 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:09.834083    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b7c5ccf8f-v7trc_bdade87f-598a-4f31-9eb7-3dcf4ef08780/copy-manifests/0.log"
Jul 29 19:59:09.854114 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:09.854081    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b7c5ccf8f-v7trc_bdade87f-598a-4f31-9eb7-3dcf4ef08780/manager/0.log"
Jul 29 19:59:10.490796 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:10.490770    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755875-7zvhh_0595a037-c243-4a25-80fb-7502bb651582/cleanup/0.log"
Jul 29 19:59:10.601974 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:10.601942    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755890-6wddw_181efb04-abbb-4abd-9dfd-5bfe475b9565/cleanup/0.log"
Jul 29 19:59:12.328357 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:12.328317    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-gqfvt_e3be0780-4663-4b6a-9ebd-26e7888aca18/manager/0.log"
Jul 29 19:59:12.881961 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:12.881926    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-cgkmb_5b8105b3-6fcf-431d-b5a0-a7b224b58387/manager/0.log"
Jul 29 19:59:13.544575 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:13.544549    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f5845fd56-v72b6_d7a34574-93af-49d6-a45d-066659d086ca/kube-auth-proxy/0.log"
Jul 29 19:59:13.889577 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:13.889548    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dhqmz_af11b487-3ae7-4c80-b7d1-1a54e4593381/payload-processing/0.log"
Jul 29 19:59:14.430085 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:14.430056    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-dgfkb_2f203259-5677-4110-8f89-9719877d36fb/main/0.log"
Jul 29 19:59:14.762940 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:14.762853    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-vckt2_998df865-bd61-46c1-a57a-de1ecc627326/main/0.log"
Jul 29 19:59:20.664112 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664071    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kxqqs/must-gather-p2zkv"]
Jul 29 19:59:20.664554 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664542    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de79cbb9-8dfb-4a36-a5dc-c592668c55bb" containerName="payload-processing"
Jul 29 19:59:20.664594 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664557    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="de79cbb9-8dfb-4a36-a5dc-c592668c55bb" containerName="payload-processing"
Jul 29 19:59:20.664594 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664568    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="689f313a-0533-4967-a237-08406df3b45b" containerName="payload-processing"
Jul 29 19:59:20.664594 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664573    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="689f313a-0533-4967-a237-08406df3b45b" containerName="payload-processing"
Jul 29 19:59:20.664594 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664586    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3" containerName="payload-pre-processing"
Jul 29 19:59:20.664594 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664593    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3" containerName="payload-pre-processing"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664602    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e264efc-fae3-459d-9a39-cf4c9f8fdae8" containerName="maas-api"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664607    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e264efc-fae3-459d-9a39-cf4c9f8fdae8" containerName="maas-api"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664613    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54adcb4d-56f8-4912-a129-eff2763aa204" containerName="payload-pre-processing"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664621    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54adcb4d-56f8-4912-a129-eff2763aa204" containerName="payload-pre-processing"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664633    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195f9ec6-3a39-4ea4-864c-4a680c23e176" containerName="payload-processing"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664639    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="195f9ec6-3a39-4ea4-864c-4a680c23e176" containerName="payload-processing"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664644    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ee2996c-86dd-405f-a049-c909121ee879" containerName="maas-api"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664649    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ee2996c-86dd-405f-a049-c909121ee879" containerName="maas-api"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664656    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341" containerName="payload-pre-processing"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664661    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341" containerName="payload-pre-processing"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664667    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06e109d8-682c-4c9a-85c0-b4f177eee884" containerName="maas-api"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664671    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="06e109d8-682c-4c9a-85c0-b4f177eee884" containerName="maas-api"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664679    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8" containerName="maas-api"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664683    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8" containerName="maas-api"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664688    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe4f538-6c3d-458b-b120-7ac00b311a39" containerName="payload-pre-processing"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664693    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe4f538-6c3d-458b-b120-7ac00b311a39" containerName="payload-pre-processing"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664700    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31" containerName="maas-api"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664713    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31" containerName="maas-api"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664722    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5edd5028-1cb7-4213-aa68-205703a7e339" containerName="payload-processing"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664727    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5edd5028-1cb7-4213-aa68-205703a7e339" containerName="payload-processing"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664733    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbf28d22-bb2e-4151-8f9c-44eb0fc714b7" containerName="payload-processing"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664737    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbf28d22-bb2e-4151-8f9c-44eb0fc714b7" containerName="payload-processing"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664744    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e029d7af-1acb-4676-af97-c16fc92d8c20" containerName="payload-pre-processing"
Jul 29 19:59:20.664752 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664750    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e029d7af-1acb-4676-af97-c16fc92d8c20" containerName="payload-pre-processing"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664814    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="519ece9f-c9f0-40ae-b28e-d8f8fd5d1d31" containerName="maas-api"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664822    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ee2996c-86dd-405f-a049-c909121ee879" containerName="maas-api"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664829    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbf28d22-bb2e-4151-8f9c-44eb0fc714b7" containerName="payload-processing"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664835    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e264efc-fae3-459d-9a39-cf4c9f8fdae8" containerName="maas-api"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664841    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce6316c-2c73-4d6b-8612-ab3d2d7cdfd8" containerName="maas-api"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664847    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbe4f538-6c3d-458b-b120-7ac00b311a39" containerName="payload-pre-processing"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664852    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5edd5028-1cb7-4213-aa68-205703a7e339" containerName="payload-processing"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664857    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c48f31a-f6d7-4672-a8e5-1c2e47c1d4d3" containerName="payload-pre-processing"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664865    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="54adcb4d-56f8-4912-a129-eff2763aa204" containerName="payload-pre-processing"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664875    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ae5ae0d-2a0c-4e9d-957a-0d24dd5df341" containerName="payload-pre-processing"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664883    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="689f313a-0533-4967-a237-08406df3b45b" containerName="payload-processing"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664888    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="de79cbb9-8dfb-4a36-a5dc-c592668c55bb" containerName="payload-processing"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664892    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e029d7af-1acb-4676-af97-c16fc92d8c20" containerName="payload-pre-processing"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664898    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="195f9ec6-3a39-4ea4-864c-4a680c23e176" containerName="payload-processing"
Jul 29 19:59:20.665475 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.664902    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="06e109d8-682c-4c9a-85c0-b4f177eee884" containerName="maas-api"
Jul 29 19:59:20.667914 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.667892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kxqqs/must-gather-p2zkv"
Jul 29 19:59:20.670991 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.670966    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kxqqs\"/\"openshift-service-ca.crt\""
Jul 29 19:59:20.670991 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.670999    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kxqqs\"/\"default-dockercfg-tgndr\""
Jul 29 19:59:20.672416 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.672393    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kxqqs\"/\"kube-root-ca.crt\""
Jul 29 19:59:20.678174 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.678147    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kxqqs/must-gather-p2zkv"]
Jul 29 19:59:20.831522 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.831484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b5d5436f-faf1-4127-af9b-a5906b8a0788-must-gather-output\") pod \"must-gather-p2zkv\" (UID: \"b5d5436f-faf1-4127-af9b-a5906b8a0788\") " pod="openshift-must-gather-kxqqs/must-gather-p2zkv"
Jul 29 19:59:20.831719 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.831523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44p5h\" (UniqueName: \"kubernetes.io/projected/b5d5436f-faf1-4127-af9b-a5906b8a0788-kube-api-access-44p5h\") pod \"must-gather-p2zkv\" (UID: \"b5d5436f-faf1-4127-af9b-a5906b8a0788\") " pod="openshift-must-gather-kxqqs/must-gather-p2zkv"
Jul 29 19:59:20.932735 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.932642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b5d5436f-faf1-4127-af9b-a5906b8a0788-must-gather-output\") pod \"must-gather-p2zkv\" (UID: \"b5d5436f-faf1-4127-af9b-a5906b8a0788\") " pod="openshift-must-gather-kxqqs/must-gather-p2zkv"
Jul 29 19:59:20.932735 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.932684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44p5h\" (UniqueName: \"kubernetes.io/projected/b5d5436f-faf1-4127-af9b-a5906b8a0788-kube-api-access-44p5h\") pod \"must-gather-p2zkv\" (UID: \"b5d5436f-faf1-4127-af9b-a5906b8a0788\") " pod="openshift-must-gather-kxqqs/must-gather-p2zkv"
Jul 29 19:59:20.933002 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.932979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b5d5436f-faf1-4127-af9b-a5906b8a0788-must-gather-output\") pod \"must-gather-p2zkv\" (UID: \"b5d5436f-faf1-4127-af9b-a5906b8a0788\") " pod="openshift-must-gather-kxqqs/must-gather-p2zkv"
Jul 29 19:59:20.942472 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.942444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44p5h\" (UniqueName: \"kubernetes.io/projected/b5d5436f-faf1-4127-af9b-a5906b8a0788-kube-api-access-44p5h\") pod \"must-gather-p2zkv\" (UID: \"b5d5436f-faf1-4127-af9b-a5906b8a0788\") " pod="openshift-must-gather-kxqqs/must-gather-p2zkv"
Jul 29 19:59:20.979221 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:20.979160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kxqqs/must-gather-p2zkv"
Jul 29 19:59:21.116235 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:21.116181    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kxqqs/must-gather-p2zkv"]
Jul 29 19:59:21.119314 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:59:21.119286    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5d5436f_faf1_4127_af9b_a5906b8a0788.slice/crio-acb6b92769de87938d1db9c160eeea66e31eae162e393ccf0de5546f9bbf75fa WatchSource:0}: Error finding container acb6b92769de87938d1db9c160eeea66e31eae162e393ccf0de5546f9bbf75fa: Status 404 returned error can't find the container with id acb6b92769de87938d1db9c160eeea66e31eae162e393ccf0de5546f9bbf75fa
Jul 29 19:59:21.121302 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:21.121281    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:59:21.474482 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:21.474448    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kxqqs/must-gather-p2zkv" event={"ID":"b5d5436f-faf1-4127-af9b-a5906b8a0788","Type":"ContainerStarted","Data":"acb6b92769de87938d1db9c160eeea66e31eae162e393ccf0de5546f9bbf75fa"}
Jul 29 19:59:22.483228 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:22.483164    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kxqqs/must-gather-p2zkv" event={"ID":"b5d5436f-faf1-4127-af9b-a5906b8a0788","Type":"ContainerStarted","Data":"f4fc682b455e8078f123e3cbf9bf7bbc41e48c2179004e6f7b386d6433fd1a20"}
Jul 29 19:59:22.483606 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:22.483242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kxqqs/must-gather-p2zkv" event={"ID":"b5d5436f-faf1-4127-af9b-a5906b8a0788","Type":"ContainerStarted","Data":"a06e619dc5192436b61b7b05238ef3214d0db4c620fd0e2e962401b7eb12bbd1"}
Jul 29 19:59:22.500140 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:22.500084    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kxqqs/must-gather-p2zkv" podStartSLOduration=1.715311502 podStartE2EDuration="2.50006623s" totalImagesPullingTime="784.754728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:22.499105731 +0000 UTC m=+3528.798778234" watchObservedRunningTime="2026-07-29 19:59:22.50006623 +0000 UTC m=+3528.799738771"
Jul 29 19:59:23.555039 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:23.555001    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gv68b_898bc211-2d04-4fe7-b981-4ce7f9c0a856/global-pull-secret-syncer/0.log"
Jul 29 19:59:23.792487 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:23.792453    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jmvb4_6b657d2b-5b8b-401b-ba94-b2c15ca3720f/konnectivity-agent/0.log"
Jul 29 19:59:23.848827 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:23.848740    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-248.ec2.internal_c90e4ebfb2b0c26370b1fc329ec87357/haproxy/0.log"
Jul 29 19:59:29.736484 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:29.736387    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-gqfvt_e3be0780-4663-4b6a-9ebd-26e7888aca18/manager/0.log"
Jul 29 19:59:30.062622 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:30.062538    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-cgkmb_5b8105b3-6fcf-431d-b5a0-a7b224b58387/manager/0.log"
Jul 29 19:59:31.956944 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:31.956904    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-vpxpc_bf81bc82-dd40-4fbb-bc69-da2c1afa8517/cluster-monitoring-operator/0.log"
Jul 29 19:59:32.098228 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.098172    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-pn52g_6ab7d3a5-50ac-4760-9486-825bbeaee987/monitoring-plugin/0.log"
Jul 29 19:59:32.133245 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.133086    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8m5pg_45473514-0e9c-4462-9784-8361953c92e4/node-exporter/0.log"
Jul 29 19:59:32.161512 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.161477    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8m5pg_45473514-0e9c-4462-9784-8361953c92e4/kube-rbac-proxy/0.log"
Jul 29 19:59:32.184645 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.184614    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8m5pg_45473514-0e9c-4462-9784-8361953c92e4/init-textfile/0.log"
Jul 29 19:59:32.369760 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.369642    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-7fwhb_9f163347-4a09-4a40-b14f-ebcf4e5f44c5/kube-rbac-proxy-main/0.log"
Jul 29 19:59:32.393802 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.393773    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-7fwhb_9f163347-4a09-4a40-b14f-ebcf4e5f44c5/kube-rbac-proxy-self/0.log"
Jul 29 19:59:32.417724 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.417691    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-7fwhb_9f163347-4a09-4a40-b14f-ebcf4e5f44c5/openshift-state-metrics/0.log"
Jul 29 19:59:32.452360 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.452309    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2f1b718a-1bde-4539-aaa0-8ce652dc5d18/prometheus/0.log"
Jul 29 19:59:32.476518 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.476482    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2f1b718a-1bde-4539-aaa0-8ce652dc5d18/config-reloader/0.log"
Jul 29 19:59:32.504049 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.504019    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2f1b718a-1bde-4539-aaa0-8ce652dc5d18/thanos-sidecar/0.log"
Jul 29 19:59:32.528650 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.528606    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2f1b718a-1bde-4539-aaa0-8ce652dc5d18/kube-rbac-proxy-web/0.log"
Jul 29 19:59:32.552108 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.552083    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2f1b718a-1bde-4539-aaa0-8ce652dc5d18/kube-rbac-proxy/0.log"
Jul 29 19:59:32.581519 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.581486    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2f1b718a-1bde-4539-aaa0-8ce652dc5d18/kube-rbac-proxy-thanos/0.log"
Jul 29 19:59:32.606097 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.606071    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2f1b718a-1bde-4539-aaa0-8ce652dc5d18/init-config-reloader/0.log"
Jul 29 19:59:32.641105 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.641073    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-mlnw2_4a07e4ea-2748-4ca4-b93c-ba5a5652f012/prometheus-operator/0.log"
Jul 29 19:59:32.664136 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.664090    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-mlnw2_4a07e4ea-2748-4ca4-b93c-ba5a5652f012/kube-rbac-proxy/0.log"
Jul 29 19:59:32.691836 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:32.691792    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-msz52_24a344c4-d14b-4181-952f-0c6aef93cb56/prometheus-operator-admission-webhook/0.log"
Jul 29 19:59:33.523254 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.523212    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"]
Jul 29 19:59:33.528006 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.527973    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.537178 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.537144    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"]
Jul 29 19:59:33.592437 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.592381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-lib-modules\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.592437 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.592434    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-podres\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.592712 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.592673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-sys\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.592781 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.592750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gcz6\" (UniqueName: \"kubernetes.io/projected/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-kube-api-access-5gcz6\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.592832 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.592813    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-proc\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.693717 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.693672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gcz6\" (UniqueName: \"kubernetes.io/projected/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-kube-api-access-5gcz6\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.693920 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.693735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-proc\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.693920 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.693792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-lib-modules\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.693920 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.693819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-podres\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.693920 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.693864    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-proc\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.694158 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.693964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-sys\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.694158 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.694011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-podres\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.694158 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.694108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-lib-modules\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.694158 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.694136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-sys\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.704481 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.704438    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gcz6\" (UniqueName: \"kubernetes.io/projected/bbe538b5-643a-4581-bff0-0f2cb60fdeb1-kube-api-access-5gcz6\") pod \"perf-node-gather-daemonset-2k6mx\" (UID: \"bbe538b5-643a-4581-bff0-0f2cb60fdeb1\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:33.843021 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:33.842914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:34.043935 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:34.043888    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"]
Jul 29 19:59:34.048154 ip-10-0-135-248 kubenswrapper[2579]: W0729 19:59:34.048078    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbbe538b5_643a_4581_bff0_0f2cb60fdeb1.slice/crio-3a27ceb537060ac9a9769cddec9c940725a7c195e04cc8a3802c702ea6c6a4ca WatchSource:0}: Error finding container 3a27ceb537060ac9a9769cddec9c940725a7c195e04cc8a3802c702ea6c6a4ca: Status 404 returned error can't find the container with id 3a27ceb537060ac9a9769cddec9c940725a7c195e04cc8a3802c702ea6c6a4ca
Jul 29 19:59:34.577608 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:34.577555    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx" event={"ID":"bbe538b5-643a-4581-bff0-0f2cb60fdeb1","Type":"ContainerStarted","Data":"90b6ce1a5a85d57a945374870e9ec33e6d23cb6ee3a2f551f5c38980d5c73182"}
Jul 29 19:59:34.578115 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:34.577630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx" event={"ID":"bbe538b5-643a-4581-bff0-0f2cb60fdeb1","Type":"ContainerStarted","Data":"3a27ceb537060ac9a9769cddec9c940725a7c195e04cc8a3802c702ea6c6a4ca"}
Jul 29 19:59:34.578115 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:34.577759    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:34.597250 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:34.597170    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx" podStartSLOduration=1.5971455159999999 podStartE2EDuration="1.597145516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:34.596886473 +0000 UTC m=+3540.896558975" watchObservedRunningTime="2026-07-29 19:59:34.597145516 +0000 UTC m=+3540.896818014"
Jul 29 19:59:34.763914 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:34.763868    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/1.log"
Jul 29 19:59:34.776429 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:34.772360    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d52sf_a622c80f-34c3-4f10-97ca-e056df858797/console-operator/2.log"
Jul 29 19:59:35.289971 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:35.289937    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-lvx6t_47785f11-d0a3-4108-a84c-71da470dca6c/download-server/0.log"
Jul 29 19:59:35.891631 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:35.891602    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-p2dkv_185b4d06-9bb4-4f51-a4e3-6a156fc3fef2/volume-data-source-validator/0.log"
Jul 29 19:59:36.905668 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:36.905635    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-79x22_15228df5-30d8-4cd8-8de4-a55d961a28d8/dns/0.log"
Jul 29 19:59:36.928813 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:36.928777    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-79x22_15228df5-30d8-4cd8-8de4-a55d961a28d8/kube-rbac-proxy/0.log"
Jul 29 19:59:37.111924 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:37.111897    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l9btm_fab1c53d-2ed6-48b8-86b6-6244542b8750/dns-node-resolver/0.log"
Jul 29 19:59:37.678990 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:37.678957    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7tb67_841e2b86-5a6c-43d7-9919-da697a9ce802/node-ca/0.log"
Jul 29 19:59:38.958476 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:38.958445    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f5845fd56-v72b6_d7a34574-93af-49d6-a45d-066659d086ca/kube-auth-proxy/0.log"
Jul 29 19:59:39.092608 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:39.092579    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dhqmz_af11b487-3ae7-4c80-b7d1-1a54e4593381/payload-processing/0.log"
Jul 29 19:59:39.968941 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:39.968914    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qkzwn_614641fc-597a-4b94-ba7c-33f078d5ed1c/serve-healthcheck-canary/0.log"
Jul 29 19:59:40.484060 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:40.484023    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-45jh7_68123628-c31f-4b00-b091-0c1e8802972a/insights-operator/1.log"
Jul 29 19:59:40.485682 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:40.485650    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-45jh7_68123628-c31f-4b00-b091-0c1e8802972a/insights-operator/0.log"
Jul 29 19:59:40.595776 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:40.595746    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2k6mx"
Jul 29 19:59:40.671794 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:40.671768    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wqsgj_acbc4c5f-87dc-474c-86a5-3d4551a92cc4/kube-rbac-proxy/0.log"
Jul 29 19:59:40.693928 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:40.693898    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wqsgj_acbc4c5f-87dc-474c-86a5-3d4551a92cc4/exporter/0.log"
Jul 29 19:59:40.719104 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:40.719069    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wqsgj_acbc4c5f-87dc-474c-86a5-3d4551a92cc4/extractor/0.log"
Jul 29 19:59:43.724451 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:43.724416    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b7c5ccf8f-v7trc_bdade87f-598a-4f31-9eb7-3dcf4ef08780/manager/0.log"
Jul 29 19:59:43.742542 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:43.742466    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b7c5ccf8f-v7trc_bdade87f-598a-4f31-9eb7-3dcf4ef08780/copy-manifests/0.log"
Jul 29 19:59:45.067164 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:45.067134    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-79b99cb5c6-df9jj_b2732d1a-d578-42da-912d-6dfa85c53344/manager/0.log"
Jul 29 19:59:45.174338 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:45.174246    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-kzc42_503e0dee-5dd3-468b-ac63-0883d7ae43b8/openshift-lws-operator/0.log"
Jul 29 19:59:45.296685 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:45.296651    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f5845fd56-v72b6_d7a34574-93af-49d6-a45d-066659d086ca/kube-auth-proxy/0.log"
Jul 29 19:59:45.397720 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:45.397684    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dhqmz_af11b487-3ae7-4c80-b7d1-1a54e4593381/payload-processing/0.log"
Jul 29 19:59:46.971682 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:46.971649    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f5845fd56-v72b6_d7a34574-93af-49d6-a45d-066659d086ca/kube-auth-proxy/0.log"
Jul 29 19:59:47.095959 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:47.095900    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dhqmz_af11b487-3ae7-4c80-b7d1-1a54e4593381/payload-processing/0.log"
Jul 29 19:59:50.739788 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:50.739693    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld_30cfb52e-9435-492b-b39d-f6ab737038bd/kube-storage-version-migrator-operator/1.log"
Jul 29 19:59:50.741426 ip-10-0-135-248 kubenswrapper[2579]: I0729 19:59:50.741397    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-wtbld_30cfb52e-9435-492b-b39d-f6ab737038bd/kube-storage-version-migrator-operator/0.log"